Multiprocessing zombie processes
in [Python]
|
From: Cameron Simpson on 24 Jul 2010 19:55 [ Please don't top post. Post below so that things read like a conversation. (And trim excess quoted junk.) It doesn't take long and makes things a lot easier for your readers. ] On 25Jul2010 04:41, Navkirat Singh <navkirats(a)gmail.com> wrote: | On 25-Jul-2010, at 4:37 AM, Navkirat Singh wrote: | > I have been meddling around with forking and multiprocessing. Now | > both of them spawn new processes from parent (atleast from what I have | > understood). I have been able to reproduce a zombie state in a fork with: | > | > import os,time | > print('before fork',os.getpid()) | > pid = os.fork() | > if pid: | > print('child: ',pid) | > time.sleep(120) | > | > Now doing a ps ax | grep <pid> I can find a zombie child process, | > but I am not being able to reproduce the same with multiprocessing. Am | > I missing something? Or multiprocessing does not have the zombie problem? | | OK I wanted zombie processes and have been able to regenerate them with multiprocessing. Now lets see how I can handle them. A zombie process is a fairly common thing and nothing to panic about usually. When you have a normal, running process it has an entry in the system process table (that "ps" shows) that points at it and shows its state. When a process exits, its exit status (exit code, whether it exited normally or from a signal, and if so which signal, some stats) _remain_ in the process table as a record of the exit status. This is shown as a "zombie" process in "ps" - a process which has exited and whose exit status is hanging around waiting to be collected. After the process' parent has issued a wait() call to collect the status the process slot if released and the :zombie" is gone. In your fork() example above, if that's the whole program, the child exits immediately and the parent has not waited for it. If the parent exits without waiting for all its children, the init process (process 1) inherits them, and it collects the statuses. Having a few zombies lying around isn't a disaster provided your program does get around to waiting for them at some point (or if your main program is short lived, so they get reaped by init after it exits). It's only a process table slot after all - little memory is needed for it. A program that spawns a lot of children and never waits for them _is_ bad in the long run. Process ids are often 2-byte integers on many systems for it's quite feasible to fill the table, and depending on the kernel's process table implementation a big process table might have other performance side effects. Cheers, -- Cameron Simpson <cs(a)zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ I'm behind a corporate Exchange Server which seems to have changed recently to converting everything it sees to HTML. How embarrassing. - Dan Espen <despen(a)telcordia.com>
From: Navkirat Singh on 24 Jul 2010 21:01 On 25-Jul-2010, at 5:25 AM, Cameron Simpson wrote: > [ Please don't top post. Post below so that things read like a > conversation. (And trim excess quoted junk.) It doesn't take long and > makes things a lot easier for your readers. ] > > On 25Jul2010 04:41, Navkirat Singh <navkirats(a)gmail.com> wrote: > | On 25-Jul-2010, at 4:37 AM, Navkirat Singh wrote: > | > I have been meddling around with forking and multiprocessing. Now > | > both of them spawn new processes from parent (atleast from what I have > | > understood). I have been able to reproduce a zombie state in a fork with: > | > > | > import os,time > | > print('before fork',os.getpid()) > | > pid = os.fork() > | > if pid: > | > print('child: ',pid) > | > time.sleep(120) > | > > | > Now doing a ps ax | grep <pid> I can find a zombie child process, > | > but I am not being able to reproduce the same with multiprocessing. Am > | > I missing something? Or multiprocessing does not have the zombie problem? > | > | OK I wanted zombie processes and have been able to regenerate them with multiprocessing. Now lets see how I can handle them. > > A zombie process is a fairly common thing and nothing to panic about > usually. > > When you have a normal, running process it has an entry in the > system process table (that "ps" shows) that points at it and shows its > state. > > When a process exits, its exit status (exit code, whether it exited > normally or from a signal, and if so which signal, some stats) _remain_ > in the process table as a record of the exit status. This is shown as a > "zombie" process in "ps" - a process which has exited and whose exit > status is hanging around waiting to be collected. > > After the process' parent has issued a wait() call to collect the status > the process slot if released and the :zombie" is gone. > > In your fork() example above, if that's the whole program, the child > exits immediately and the parent has not waited for it. > > If the parent exits without waiting for all its children, the init > process (process 1) inherits them, and it collects the statuses. > > Having a few zombies lying around isn't a disaster provided your program > does get around to waiting for them at some point (or if your main > program is short lived, so they get reaped by init after it exits). > It's only a process table slot after all - little memory is needed for > it. > > A program that spawns a lot of children and never waits for them _is_ > bad in the long run. Process ids are often 2-byte integers on many > systems for it's quite feasible to fill the table, and depending on the > kernel's process table implementation a big process table might have other > performance side effects. > > Cheers, > -- > Cameron Simpson <cs(a)zip.com.au> DoD#743 > http://www.cskk.ezoshosting.com/cs/ > > I'm behind a corporate Exchange Server which seems to have > changed recently to converting everything it sees to HTML. > How embarrassing. - Dan Espen <despen(a)telcordia.com> Thanks a lot for all the information. It cleared a lot of my doubts. Nav
From: Lawrence D'Oliveiro on 26 Jul 2010 06:30 In message <mailman.1127.1280014712.1673.python-list(a)python.org>, Chris Rebert wrote: > "Paging Dr. Frankenstein. Dr. Frankenstein to the lab. Paging Dr. > Frankenstein." > > Most people try to /avoid/ making zombies. Is there some connection between Frankenstein and zombies?
From: Chris Rebert on 26 Jul 2010 06:40 On Mon, Jul 26, 2010 at 3:30 AM, Lawrence D'Oliveiro <ldo(a)geek-central.gen.new_zealand> wrote: > In message <mailman.1127.1280014712.1673.python-list(a)python.org>, Chris > Rebert wrote: > >> "Paging Dr. Frankenstein. Dr. Frankenstein to the lab. Paging Dr. >> Frankenstein." >> >> Most people try to /avoid/ making zombies. > > Is there some connection between Frankenstein and zombies? His monster is zombie-ish; it's made of reanimated corpse parts. It's also mentioned in Wikipedia's "Zombie" article as influencing future works about the undead. Cheers, Chris -- Not a perfect comedic reference, I admit.
From: Michele Simionato on 26 Jul 2010 08:03
On Jul 25, 1:11 am, Navkirat Singh <navkir...(a)gmail.com> wrote: > OK I wanted zombie processes and have been able to regenerate them with multiprocessing. Now lets see how I can handle them. The multiprocessing docs say: """ Joining zombie processes On Unix when a process finishes but has not been joined it becomes a zombie. There should never be very many because each time a new process starts (or active_children() is called) all completed processes which have not yet been joined will be joined. Also calling a finished processs Process.is_alive() will join the process. Even so it is probably good practice to explicitly join all the processes that you start. """ |