Prev: Ruby Midwest Conference Opens Call for Speakers
Next: unsubscribe
From: Joe Martin on 5 Mar 2010 08:06
Robert Klemme wrote:
> 2010/3/4 Joe Martin <jm202(a)yahoo.com>:
>> check how many threads are running at once, and if the number of running
>> threads falls below the limit of 5, it takes the next thread out of the
>> pool and runs it? �Not sure how I would go about doing this, pretty new
>> to multithreading.
>
> Why do you create a pool much larger than the load you want to accept?
> Usually the pool size is used to limit concurrency. Actually that is
> the main purpose of thread pools.
>
> If you have different tasks for which you want to have different
> limits on concurrency you could also create several pools with
> different sizes.
>
> Kind regards
>
> robert

When running this program, I will provide a list of items that need
processing. In some cases, this list can be as long as 250 items, in
other cases well over 50,000. The processing of each item can take
anywhere from 15 to 60 seconds per item, so you can see there is a
benefit to multithreading here. In processing each item, there are also
a number of database calls that occur, so I would like to put a cap on
the number of actively running threads to avoid overwhelming the
database. Am I going about this the wrong way? Is there a more
effecient more suitable way of doing this?
--
Posted via http://www.ruby-forum.com/.

From: Robert Klemme on 5 Mar 2010 08:15
2010/3/5 Joe Martin <jm202(a)yahoo.com>:
> Robert Klemme wrote:
>> 2010/3/4 Joe Martin <jm202(a)yahoo.com>:
>>> check how many threads are running at once, and if the number of running
>>> threads falls below the limit of 5, it takes the next thread out of the
>>> pool and runs it? �Not sure how I would go about doing this, pretty new
>>> to multithreading.
>>
>> Why do you create a pool much larger than the load you want to accept?
>>  Usually the pool size is used to limit concurrency.  Actually that is
>> the main purpose of thread pools.
>>
>> If you have different tasks for which you want to have different
>> limits on concurrency you could also create several pools with
>> different sizes.
>>
>> Kind regards
>>
>> robert
>
> When running this program, I will provide a list of items that need
> processing.  In some cases, this list can be as long as 250 items, in
> other cases well over 50,000.  The processing of each item can take
> anywhere from 15 to 60 seconds per item, so you can see there is a
> benefit to multithreading here.  In processing each item, there are also
> a number of database calls that occur, so I would like to put a cap on
> the number of actively running threads to avoid overwhelming the
> database.  Am I going about this the wrong way?  Is there a more
> effecient more suitable way of doing this?

For this scenario a thread pool with fixed size seems sufficient.

queue = Queue.new # or bounded queue

def cont(item) !item.nil? end

threads = (1..10).map do
Thread.new do
while (cont(item = queue.deq))
# .. process
end
end
end

queue.enc "Task"

threads.size.times do
queue.enq nil # terminate
end

threads.each {|th| th.join}

As simple as that.

Kind regards

robert


--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

From: Chuck Remes on 5 Mar 2010 08:48
On Mar 5, 2010, at 7:06 AM, Joe Martin wrote:

> Robert Klemme wrote:
>> 2010/3/4 Joe Martin <jm202(a)yahoo.com>:
>>> check how many threads are running at once, and if the number of running
>>> threads falls below the limit of 5, it takes the next thread out of the
>>> pool and runs it? �Not sure how I would go about doing this, pretty new
>>> to multithreading.
>>
>> Why do you create a pool much larger than the load you want to accept?
>> Usually the pool size is used to limit concurrency. Actually that is
>> the main purpose of thread pools.
>>
>> If you have different tasks for which you want to have different
>> limits on concurrency you could also create several pools with
>> different sizes.
>>
>> Kind regards
>>
>> robert
>
> When running this program, I will provide a list of items that need
> processing. In some cases, this list can be as long as 250 items, in
> other cases well over 50,000. The processing of each item can take
> anywhere from 15 to 60 seconds per item, so you can see there is a
> benefit to multithreading here. In processing each item, there are also
> a number of database calls that occur, so I would like to put a cap on
> the number of actively running threads to avoid overwhelming the
> database. Am I going about this the wrong way? Is there a more
> effecient more suitable way of doing this?

The Threadz gem let's you create a thread pool and then wait on its completion before you add more to it. This mechanism will help you cap the number of threads making database calls.

cr


From: Joe Martin on 5 Mar 2010 15:59
Robert Klemme wrote:
> For this scenario a thread pool with fixed size seems sufficient.
>

Very good. This works quite nicely as well.

Just wondering, are there any performance benefits of using one method
over the other?
--
Posted via http://www.ruby-forum.com/.

From: Caleb Clausen on 5 Mar 2010 16:52
On 3/5/10, Joe Martin <jm202(a)yahoo.com> wrote:
> Robert Klemme wrote:
>> For this scenario a thread pool with fixed size seems sufficient.
>>
>
> Very good. This works quite nicely as well.
>
> Just wondering, are there any performance benefits of using one method
> over the other?

Yes. Threads use memory (quite a lot of it, in fact). Mostly this goes
to the thread's stack. Limiting the number of threads saves quite a
bit of memory. I'm not sure there's any improvement in the amount of
cpu time either way, other than perhaps some fewer cache misses
resulting from using less memory.

First  |  Prev  |  Next  |  Last
Pages: 1 2 3
Prev: Ruby Midwest Conference Opens Call for Speakers
Next: unsubscribe