Prev: Open an explorer window
Next: Thread error "undefined method `keys' for nil:NilClass"
From: Glen Holcomb on 29 Oct 2009 16:30
On Thu, Oct 29, 2009 at 2:04 PM, Tony Arcieri <tony(a)medioh.com> wrote:

> On Thu, Oct 29, 2009 at 11:48 AM, Glen Holcomb <damnbigman(a)gmail.com>
> wrote:
>
> > You are going to want Ruby 1.9 for this. In 1.8 threads are "green",
> > basically they only exists as threads inside the VM so you still only hit
> > one core and any blocking system I/O will block all of your threads.
> >
>
> Ruby 1.9 isn't going to help you when using threads to distribute
> computation across CPU cores. The Global VM Lock ensures that simultaneous
> computation is still limited to one core.
>
> JRuby, on the other hand, does not have this limitation. On MRI/1.9 I
> would
> recommend using multiple processes.
>
> --
> Tony Arcieri
> Medioh/Nagravision
>

Ah, I did not know that.

--
"Hey brother Christian with your high and mighty errand, Your actions speak
so loud, I can’t hear a word you’re saying."

-Greg Graffin (Bad Religion)

From: Robert Klemme on 29 Oct 2009 18:03
On 10/29/2009 09:04 PM, Tony Arcieri wrote:

> On Thu, Oct 29, 2009 at 11:48 AM, Glen Holcomb <damnbigman(a)gmail.com> wrote:
>
>> You are going to want Ruby 1.9 for this. In 1.8 threads are "green",
>> basically they only exists as threads inside the VM so you still only hit
>> one core and any blocking system I/O will block all of your threads.
>
> Ruby 1.9 isn't going to help you when using threads to distribute
> computation across CPU cores. The Global VM Lock ensures that simultaneous
> computation is still limited to one core.

Are you saying that the global VM lock even extends to several
processes? Because Marc did not want to use threads for distribution
but rather processes.

Kind regards

robert

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/
From: Robert Klemme on 29 Oct 2009 18:21
On 10/29/2009 03:56 PM, Marc Hoeppner wrote:
> Hi,
>
> I've been reading around a bit but couldn't find a solution that worked,
> so here goes:
>
> I am running ruby 1.8 and want to make full use of a quad core CPU
> (64bit, Ubuntu) in a task that lends itself to multithreading/multicore
> use.
>
> It's basically an array of objects that are each use in a fairly CPU
> intensive job, so I figured I could have 4 of them run at the same time
> , one on each CPU.
>
> BUT...
>
> The only reasonably understandably suggestion looked something like:
>
> ----
> threads = 4
> my_array = [something_here]
>
> threads.times do
> Process.fork(a_method(my_array.shift))
> end
>
> my_array.each do |object|
> Process.wait(0)
> Process.fork(a_method(object))
> end
> ---
>
> But this still only used one CPU (and looks a bit ugly..). Is that some
> limitation of ruby (v 1.8 specifically) or am I doing something wrong?

I believe you are not using Process.fork properly. In fact, I am
surprised that you do not get an exception:

irb(main):001:0> Process.fork("foo")
ArgumentError: wrong number of arguments (1 for 0)
from (irb):1:in `fork'
from (irb):1
from :0

Basically what you do is you do a calculation (a_method(object)) and
_then_ you create a process. No surprise that only one CPU is busy.

Here's something else that you could do

processes = 4

my_array.each_slice my_array.size / processes do |tasks|
fork do
tasks.each do |task|
a_method(task)
end
end
end

Process.waitall

Drawback is that one of those processes might accidentally get all the
easy tasks and you do not utilize CPUs optimally. Here's another
solution that does not have that issue

processes = 4
count = 0

my_array.each do |task|
if count == processes
Process.wait
count -= 1
end

fork do
a_method(task)
end
count += 1
end

Process.waitall

You can see that it works with this example:

processes = 4
count = 0

10.times do |task|
if count == processes
Process.wait
count -= 1
end

fork do
printf "%-20s start %4d %4d\n", Time.now, $$, task
sleep rand(5) + 2
printf "%-20s end %4d %4d\n", Time.now, $$, task
end
count += 1
end

Process.waitall


Kind regards

robert

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/
From: Tony Arcieri on 29 Oct 2009 18:25
[Note: parts of this message were removed to make it a legal post.]

On Thu, Oct 29, 2009 at 4:05 PM, Robert Klemme
<shortcutter(a)googlemail.com>wrote:

> On 10/29/2009 09:04 PM, Tony Arcieri wrote:
>
>> Ruby 1.9 isn't going to help you when using threads to distribute
>> computation across CPU cores. The Global VM Lock ensures that
>> simultaneous
>> computation is still limited to one core.
>>
>
> Are you saying that the global VM lock even extends to several processes?
> Because Marc did not want to use threads for distribution but rather
> processes.
>

No, if you look over my post again it specifically mentions the GVL applies
to threads and suggests using processes.

--
Tony Arcieri
Medioh/Nagravision

From: Robert Klemme on 30 Oct 2009 04:06
2009/10/29 Tony Arcieri <tony(a)medioh.com>:
> On Thu, Oct 29, 2009 at 4:05 PM, Robert Klemme
> <shortcutter(a)googlemail.com>wrote:
>
>> On 10/29/2009 09:04 PM, Tony Arcieri wrote:
>>
>>> Ruby 1.9 isn't going to help you when using threads to distribute
>>> computation across CPU cores.  The Global VM Lock ensures that
>>> simultaneous
>>> computation is still limited to one core.
>>
>> Are you saying that the global VM lock even extends to several processes?
>>  Because Marc did not want to use threads for distribution but rather
>> processes.
>
> No, if you look over my post again it specifically mentions the GVL applies
> to threads and suggests using processes.

I figured as much. The thread discussion does not help Marc, because
he explicitly wanted to use processes for core utilization. Basically
Glen sent us in the wrong direction though. :-)

Cheers

robert


--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

First  |  Prev  |  Next  |  Last
Pages: 1 2 3 4
Prev: Open an explorer window
Next: Thread error "undefined method `keys' for nil:NilClass"