Limit number of concurrent running threads in pool
in [Ruby]
|
From: Joe Martin on 4 Mar 2010 13:49 Hi I created a pool of threads (say, 500 threads) to process. However, due to the weight of each thread, I want to limit the number of threads that run concurrently. So how would I go about putting a limit on the number of threads that run at any given time? I would like to take, say, 5 threads from the pool and run them, and as each one completes, it is removed from the pool and is replaced with a new thread from the pool. Could this be done with a "spy" thread, in that it constantly loops to check how many threads are running at once, and if the number of running threads falls below the limit of 5, it takes the next thread out of the pool and runs it? Not sure how I would go about doing this, pretty new to multithreading. Thanks! -- Posted via http://www.ruby-forum.com/.
From: Roger Pack on 4 Mar 2010 14:05 > I created a pool of threads (say, 500 threads) to process. However, due > to the weight of each thread, I want to limit the number of threads that > run concurrently. > > So how would I go about putting a limit on the number of threads that > run at any given time? I would like to take, say, 5 threads from the > pool and run them, and as each one completes, it is removed from the > pool and is replaced with a new thread from the pool. http://github.com/spox/actionpool might help. -r -- Posted via http://www.ruby-forum.com/.
From: Joe Martin on 4 Mar 2010 17:14 Roger Pack wrote: >> I created a pool of threads (say, 500 threads) to process. However, due >> to the weight of each thread, I want to limit the number of threads that >> run concurrently. >> >> So how would I go about putting a limit on the number of threads that >> run at any given time? I would like to take, say, 5 threads from the >> pool and run them, and as each one completes, it is removed from the >> pool and is replaced with a new thread from the pool. > > http://github.com/spox/actionpool > > might help. > -r Thank you very much, Roger. I found this link (http://snippets.dzone.com/posts/show/3276) which it looks like you had a part in as well and just got that code working shortly after posting this thread. But looking into ActionPool, it definitely offers expanded functionality so I will probably implement that solution instead. Cheers! -- Posted via http://www.ruby-forum.com/.
From: Robert Klemme on 5 Mar 2010 07:26 2010/3/4 Joe Martin <jm202(a)yahoo.com>: > I created a pool of threads (say, 500 threads) to process. However, due > to the weight of each thread, I want to limit the number of threads that > run concurrently. > > So how would I go about putting a limit on the number of threads that > run at any given time? I would like to take, say, 5 threads from the > pool and run them, and as each one completes, it is removed from the > pool and is replaced with a new thread from the pool. > > Could this be done with a "spy" thread, in that it constantly loops to > check how many threads are running at once, and if the number of running > threads falls below the limit of 5, it takes the next thread out of the > pool and runs it? Not sure how I would go about doing this, pretty new > to multithreading. Why do you create a pool much larger than the load you want to accept? Usually the pool size is used to limit concurrency. Actually that is the main purpose of thread pools. If you have different tasks for which you want to have different limits on concurrency you could also create several pools with different sizes. Kind regards robert -- remember.guy do |as, often| as.you_can - without end http://blog.rubybestpractices.com/
From: Chuck Remes on 5 Mar 2010 07:59
On Mar 4, 2010, at 12:49 PM, Joe Martin wrote: > Hi > I created a pool of threads (say, 500 threads) to process. However, due > to the weight of each thread, I want to limit the number of threads that > run concurrently. > > So how would I go about putting a limit on the number of threads that > run at any given time? I would like to take, say, 5 threads from the > pool and run them, and as each one completes, it is removed from the > pool and is replaced with a new thread from the pool. > > Could this be done with a "spy" thread, in that it constantly loops to > check how many threads are running at once, and if the number of running > threads falls below the limit of 5, it takes the next thread out of the > pool and runs it? Not sure how I would go about doing this, pretty new > to multithreading. I've had very good success using the Threadz gem. http://github.com/nanodeath/threadz It's quite easy to understand and works very well with MRI and JRuby. cr |