freebsd and multiprocessing
in [Python]
|
Prev: Queue peek?
Next: Broken references in postings
From: Tim Arnold on 2 Mar 2010 11:31 Hi, I'm intending to use multiprocessing on a freebsd machine (6.3 release, quad core, 8cpus, amd64). I see in the doc that on this platform I can't use synchronize: ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770. As far as I can tell, I have no need to synchronize the processes--I have several processes run separately and I need to know when they're all finished; there's no communication between them and each owns its own log file for output. Is anyone using multiprocessing on FreeBSD and run into any other gotchas? thanks, --Tim Arnold
From: Philip Semanchuk on 2 Mar 2010 11:52 On Mar 2, 2010, at 11:31 AM, Tim Arnold wrote: > Hi, > I'm intending to use multiprocessing on a freebsd machine (6.3 > release, quad core, 8cpus, amd64). I see in the doc that on this > platform I can't use synchronize: > > ImportError: This platform lacks a functioning sem_open > implementation, therefore, the required synchronization primitives > needed will not function, see issue 3770. > > As far as I can tell, I have no need to synchronize the processes--I > have several processes run separately and I need to know when they're > all finished; there's no communication between them and each owns its > own log file for output. > > Is anyone using multiprocessing on FreeBSD and run into any other > gotchas? Hi Tim, I don't use multiprocessing but I've written two low-level IPC packages, one for SysV IPC and the other for POSIX IPC. I think that multiprocessing prefers POSIX IPC (which is where sem_open() comes from). I don't know what it uses if that's not available, but SysV IPC seems a likely alternative. I must emphasize, however, that that's a guess on my part. FreeBSD didn't have POSIX IPC support until 7.0, and that was sort of broken until 7.2. As it happens, I was testing my POSIX IPC code against 7.2 last night and it works just fine. SysV IPC works under FreeBSD 6 (and perhaps earlier versions; 6 is the oldest I've tested). ISTR that by default each message queue is limited to 2048 bytes in total size. 'sysctl kern.ipc' can probably tell you that and may even let you change it. Other than that I can't think of any SysV limitations that might bite you. HTH Philip
From: Tim Arnold on 2 Mar 2010 12:59 On Mar 2, 11:52 am, Philip Semanchuk <phi...(a)semanchuk.com> wrote: > On Mar 2, 2010, at 11:31 AM, Tim Arnold wrote: > > > > > > > Hi, > > I'm intending to use multiprocessing on a freebsd machine (6.3 > > release, quad core, 8cpus, amd64). I see in the doc that on this > > platform I can't use synchronize: > > > ImportError: This platform lacks a functioning sem_open > > implementation, therefore, the required synchronization primitives > > needed will not function, see issue 3770. > > > As far as I can tell, I have no need to synchronize the processes--I > > have several processes run separately and I need to know when they're > > all finished; there's no communication between them and each owns its > > own log file for output. > > > Is anyone using multiprocessing on FreeBSD and run into any other > > gotchas? > > Hi Tim, > I don't use multiprocessing but I've written two low-level IPC > packages, one for SysV IPC and the other for POSIX IPC. > > I think that multiprocessing prefers POSIX IPC (which is where > sem_open() comes from). I don't know what it uses if that's not > available, but SysV IPC seems a likely alternative. I must emphasize, > however, that that's a guess on my part. > > FreeBSD didn't have POSIX IPC support until 7.0, and that was sort of > broken until 7.2. As it happens, I was testing my POSIX IPC code > against 7.2 last night and it works just fine. > > SysV IPC works under FreeBSD 6 (and perhaps earlier versions; 6 is the > oldest I've tested). ISTR that by default each message queue is > limited to 2048 bytes in total size. 'sysctl kern.ipc' can probably > tell you that and may even let you change it. Other than that I can't > think of any SysV limitations that might bite you. > > HTH > Philip Hi Philip, Thanks for that information. I wish I could upgrade the machine to 7.2! alas, out of my power. I get the following results from sysctl: % sysctl kern.ipc | grep msg kern.ipc.msgseg: 2048 kern.ipc.msgssz: 8 kern.ipc.msgtql: 40 kern.ipc.msgmnb: 2048 kern.ipc.msgmni: 40 kern.ipc.msgmax: 16384 I'll write some test programs using multiprocessing and see how they go before committing to rewrite my current code. I've also been looking at 'parallel python' although it may have the same issues. http://www.parallelpython.com/ thanks again, --Tim
From: Tim Arnold on 2 Mar 2010 13:31 On Mar 2, 12:59 pm, Tim Arnold <a_j...(a)bellsouth.net> wrote: > On Mar 2, 11:52 am, Philip Semanchuk <phi...(a)semanchuk.com> wrote: > > On Mar 2, 2010, at 11:31 AM, Tim Arnold wrote: > > > > Hi, > > > I'm intending to use multiprocessing on a freebsd machine (6.3 > > > release, quad core, 8cpus, amd64). I see in the doc that on this > > > platform I can't use synchronize: > > > > ImportError: This platform lacks a functioning sem_open > > > implementation, therefore, the required synchronization primitives > > > needed will not function, see issue 3770. > > > > As far as I can tell, I have no need to synchronize the processes--I > > > have several processes run separately and I need to know when they're > > > all finished; there's no communication between them and each owns its > > > own log file for output. > > > > Is anyone using multiprocessing on FreeBSD and run into any other > > > gotchas? > > > Hi Tim, > > I don't use multiprocessing but I've written two low-level IPC > > packages, one for SysV IPC and the other for POSIX IPC. > > > I think that multiprocessing prefers POSIX IPC (which is where > > sem_open() comes from). I don't know what it uses if that's not > > available, but SysV IPC seems a likely alternative. I must emphasize, > > however, that that's a guess on my part. > > > FreeBSD didn't have POSIX IPC support until 7.0, and that was sort of > > broken until 7.2. As it happens, I was testing my POSIX IPC code > > against 7.2 last night and it works just fine. > > > SysV IPC works under FreeBSD 6 (and perhaps earlier versions; 6 is the > > oldest I've tested). ISTR that by default each message queue is > > limited to 2048 bytes in total size. 'sysctl kern.ipc' can probably > > tell you that and may even let you change it. Other than that I can't > > think of any SysV limitations that might bite you. > > > HTH > > Philip > > Hi Philip, > Thanks for that information. I wish I could upgrade the machine to > 7.2! alas, out of my power. I get the following results from sysctl: > % sysctl kern.ipc | grep msg > kern.ipc.msgseg: 2048 > kern.ipc.msgssz: 8 > kern.ipc.msgtql: 40 > kern.ipc.msgmnb: 2048 > kern.ipc.msgmni: 40 > kern.ipc.msgmax: 16384 > > I'll write some test programs using multiprocessing and see how they > go before committing to rewrite my current code. I've also been > looking at 'parallel python' although it may have the same issues.http://www.parallelpython.com/ > > thanks again, > --Tim Well that didn't work out well. I can't import either Queue or Pool from multiprocessing, so I'm back to the drawing board. I'll see now how parallel python does on freebsd. --Tim
From: Philip Semanchuk on 2 Mar 2010 14:28
On Mar 2, 2010, at 1:31 PM, Tim Arnold wrote: > On Mar 2, 12:59 pm, Tim Arnold <a_j...(a)bellsouth.net> wrote: >> On Mar 2, 11:52 am, Philip Semanchuk <phi...(a)semanchuk.com> wrote: >>> On Mar 2, 2010, at 11:31 AM, Tim Arnold wrote: >> >>>> Hi, >>>> I'm intending to use multiprocessing on a freebsd machine (6.3 >>>> release, quad core, 8cpus, amd64). I see in the doc that on this >>>> platform I can't use synchronize: >> >>>> ImportError: This platform lacks a functioning sem_open >>>> implementation, therefore, the required synchronization primitives >>>> needed will not function, see issue 3770. >> >>>> As far as I can tell, I have no need to synchronize the >>>> processes--I >>>> have several processes run separately and I need to know when >>>> they're >>>> all finished; there's no communication between them and each owns >>>> its >>>> own log file for output. >> >>>> Is anyone using multiprocessing on FreeBSD and run into any other >>>> gotchas? >> >>> Hi Tim, >>> I don't use multiprocessing but I've written two low-level IPC >>> packages, one for SysV IPC and the other for POSIX IPC. >> >>> I think that multiprocessing prefers POSIX IPC (which is where >>> sem_open() comes from). I don't know what it uses if that's not >>> available, but SysV IPC seems a likely alternative. I must >>> emphasize, >>> however, that that's a guess on my part. >> >>> FreeBSD didn't have POSIX IPC support until 7.0, and that was sort >>> of >>> broken until 7.2. As it happens, I was testing my POSIX IPC code >>> against 7.2 last night and it works just fine. >> >>> SysV IPC works under FreeBSD 6 (and perhaps earlier versions; 6 is >>> the >>> oldest I've tested). ISTR that by default each message queue is >>> limited to 2048 bytes in total size. 'sysctl kern.ipc' can probably >>> tell you that and may even let you change it. Other than that I >>> can't >>> think of any SysV limitations that might bite you. >> >>> HTH >>> Philip >> >> Hi Philip, >> Thanks for that information. I wish I could upgrade the machine to >> 7.2! alas, out of my power. I get the following results from sysctl: >> % sysctl kern.ipc | grep msg >> kern.ipc.msgseg: 2048 >> kern.ipc.msgssz: 8 >> kern.ipc.msgtql: 40 >> kern.ipc.msgmnb: 2048 >> kern.ipc.msgmni: 40 >> kern.ipc.msgmax: 16384 >> >> I'll write some test programs using multiprocessing and see how they >> go before committing to rewrite my current code. I've also been >> looking at 'parallel python' although it may have the same issues.http://www.parallelpython.com/ >> >> thanks again, >> --Tim > > Well that didn't work out well. I can't import either Queue or Pool > from multiprocessing, so I'm back to the drawing board. I'll see now > how parallel python does on freebsd. Sorry to hear that didn't work for you. Should you need to get down to the nuts & bolts level, my module for SysV IPC is here: http://semanchuk.com/philip/sysv_ipc/ Good luck with Parallel Python, Philip |