Possible to make subprocess.Popen jobs run serially rather thanin parallel?
in [Python]
|
Prev: Possible to make subprocess.Popen jobs run serially rather than in parallel?
Next: Introducing - Pyaudiogen
From: Stephen Hansen on 15 Jun 2010 15:03 On 6/15/10 11:52 AM, Chris Seberino wrote: > Possible to make subprocess.Popen jobs run serially rather than in > parallel? > > In other words, if a computer is low on memory and doesn't mind > waiting.....can Popen be configured to submit to a queue and run jobs > *ONE AT TIME*?? > > That might be useful and avoid crashes and disk swapping. Just call "process.wait()" after you call process = subprocess.Popen(...) -- Stephen Hansen ... Also: Ixokai ... Mail: me+list/python (AT) ixokai (DOT) io ... Blog: http://meh.ixokai.io/
From: Thomas Jollans on 16 Jun 2010 10:13 On 06/16/2010 04:04 PM, Chris Seberino wrote: > On Jun 15, 2:03 pm, Stephen Hansen <me+list/pyt...(a)ixokai.io> wrote: > >> Just call "process.wait()" after you call process = subprocess.Popen(...) > > I may have not been clear..... > I *don't* want web app to block on Popen.wait. > I *do* want the Popen process to run in the background which web app > still runs doing other things. > > Rather, I don't want *MANY* Popen processes to run in the > background....just one preferably. so create a single extra worker process. Main app spawns or connects to worker process, worker process takes care of getting the work done, either by forking and waiting (sounds silly to me...) or by just doing the work it's asked to so. How about threads? How about using os.fork instead of subprocess? How about using the new multiprocessing module?
From: Jean-Michel Pichavant on 16 Jun 2010 10:45 Chris Seberino wrote: > On Jun 15, 2:03 pm, Stephen Hansen <me+list/pyt...(a)ixokai.io> wrote: > > >> Just call "process.wait()" after you call process = subprocess.Popen(...) >> > > I may have not been clear..... > I *don't* want web app to block on Popen.wait. > I *do* want the Popen process to run in the background which web app > still runs doing other things. > > Rather, I don't want *MANY* Popen processes to run in the > background....just one preferably. > > cs > then put the process.wait() in a single thread, this thread will chain Popen calls in the background. If your executing python code in your Popen close, you'd better go for the multiprocessing module (python 2.6+) JM
From: Stephen Hansen on 16 Jun 2010 12:27
On 6/16/10 7:04 AM, Chris Seberino wrote: > On Jun 15, 2:03 pm, Stephen Hansen <me+list/pyt...(a)ixokai.io> wrote: > >> Just call "process.wait()" after you call process = subprocess.Popen(...) > > I may have not been clear..... > I *don't* want web app to block on Popen.wait. > I *do* want the Popen process to run in the background which web app > still runs doing other things. > > Rather, I don't want *MANY* Popen processes to run in the > background....just one preferably. The simpliest method that comes to mind then is to have a "Process Runner" thread that you start when the web app begins. Then create a Queue.Queue() instance, share it between said thread and your web app. When you want to run an application, do Queue.put( (argspec,) ) Have Process Runner do a blocking wait with Queue.get(). When you wake it up with Queue.put, have it pass the args off into subprocess.Popen. Then have it do process.wait() to block on said process's completion. Once it's done, our little infinite loop jumps to the top, and it calls queue.get() again -- if another process request has been put in, it immediately gets it and goes and runs it, thus your processes are executing one at a time. If nothing is ready for it, it blocks until you wake it up. Something like (written off of top of head, may have errors): import threading import Queue import subprocess class ProcessRunner(threading.Thread): def __init__(self, queue): self._queue = queue self.setDaemon(True) def run(self): while True: args, kwargs = self._queue.get() process = subprocess.Popen(*args, **kwargs) process.wait() # ... And somewhere in our real web-app initialization, we do... runner_queue = Queue.Queue() runner_thread = ProcessRunner(runner_queue) runner_thread.start() # ... And later, when we want to start a process ... runner_queue.put( (("ls -la",), {"shell": False}) ) # (*) see bottom -- Stephen Hansen ... Also: Ixokai ... Mail: me+list/python (AT) ixokai (DOT) io ... Blog: http://meh.ixokai.io/ P.S. Passing in 'args' and 'kwargs' into the queue is usually in my experience overkill (in addition to being slightly ugly); generally the subprocesses I want to run are similar in nature or environment, so I just have the runner-thread smart. But, the above is the most naive implementation. |