Which multiprocessing methods use shared memory?
in [Python]
|
Prev: Why are String Formatted Queries Considered So Magical? (Spammeranalysis)
Next: tkinter unicode question
From: Kevin Ar18 on 27 Jul 2010 15:12 I'm not sure my previous message went through (I wasn't subscribe), so I'm gonna try again. The multiprocessing module has 4 methods for sharing data between processes: Queues Pipes Shared Memory Map Server Process Which of these use shared memory? I understand that the 3rd (Shared Memory Map) does, but what about Queues? Thanks, Kevin _________________________________________________________________ Hotmail is redefining busy with tools for the New Busy. Get more from your inbox. http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_2
From: MRAB on 27 Jul 2010 15:30 Kevin Ar18 wrote: > I'm not sure my previous message went through (I wasn't subscribe), so I'm gonna try again. > > The multiprocessing module has 4 methods for sharing data between processes: > Queues > Pipes > Shared Memory Map > Server Process > > Which of these use shared memory? > > I understand that the 3rd (Shared Memory Map) does, but what about Queues? > The documentation says: """class multiprocessing.Queue([maxsize]) Returns a process shared queue implemented using a pipe and a few locks/semaphores. When a process first puts an item on the queue a feeder thread is started which transfers objects from a buffer into the pipe."""
From: John Nagle on 28 Jul 2010 01:26
On 7/27/2010 12:30 PM, MRAB wrote: > Kevin Ar18 wrote: >> I'm not sure my previous message went through (I wasn't subscribe), so >> I'm gonna try again. >> >> The multiprocessing module has 4 methods for sharing data between >> processes: >> Queues >> Pipes >> Shared Memory Map >> Server Process >> >> Which of these use shared memory? >> >> I understand that the 3rd (Shared Memory Map) does, but what about >> Queues? >> > The documentation says: > > """class multiprocessing.Queue([maxsize]) > Returns a process shared queue implemented using a pipe and a few > locks/semaphores. When a process first puts an item on the queue a > feeder thread is started which transfers objects from a buffer into the > pipe.""" Python's "shared memory" between processes is un-Pythonic. You can't share Python object, only C objects. The locking mechanisms are very limited, and slow; locking actually takes place via messages over pipes. There's no dynamic allocation in the shared area. Realistically, if you're using Python, you're not that concerned about compute speed. So don't bother with shared memory, which is a performance optimization. John Nagle |