Using multicore CPUs in parallel tasks
in [Ruby]
|
From: Marc Hoeppner on 29 Oct 2009 10:56 Hi, I've been reading around a bit but couldn't find a solution that worked, so here goes: I am running ruby 1.8 and want to make full use of a quad core CPU (64bit, Ubuntu) in a task that lends itself to multithreading/multicore use. It's basically an array of objects that are each use in a fairly CPU intensive job, so I figured I could have 4 of them run at the same time , one on each CPU. BUT... The only reasonably understandably suggestion looked something like: ---- threads = 4 my_array = [something_here] threads.times do Process.fork(a_method(my_array.shift)) end my_array.each do |object| Process.wait(0) Process.fork(a_method(object)) end --- But this still only used one CPU (and looks a bit ugly..). Is that some limitation of ruby (v 1.8 specifically) or am I doing something wrong? Cheers, Marc -- Posted via http://www.ruby-forum.com/.
From: Rajinder Yadav on 29 Oct 2009 12:36 On Thu, Oct 29, 2009 at 10:56 AM, Marc Hoeppner <marc.hoeppner(a)molbio.su.se> wrote: > Hi, > > I've been reading around a bit but couldn't find a solution that worked, > so here goes: > > I am running ruby 1.8 and want to make full use of a quad core CPU > (64bit, Ubuntu) in a task that lends itself to multithreading/multicore > use. > > It's basically an array of objects that are each use in a fairly CPU > intensive job, so I figured I could have 4 of them run at the same time > , one on each CPU. You might want to checkout Pure and Tiamat and talk to James Lawrence (see links). He seems to have something you are asking for. I don't know much about these 2 project, they came by my radar a few days ago but I think it's cool what James is working on! == Links * Pure: http://purefunctional.rubyforge.org * Documentation: http://tiamat.rubyforge.org * Download: http://rubyforge.org/frs/?group_id=9145 * Rubyforge home: http://rubyforge.org/projects/tiamat * Repository: http://github.com/quix/tiamat == Author * James M. Lawrence <quixoticsycophant(a)gmail.com> > BUT... > > The only reasonably understandably suggestion looked something like: > > ---- > threads = 4 > my_array = [something_here] > > threads.times do > Process.fork(a_method(my_array.shift)) > end > > my_array.each do |object| > Process.wait(0) > Process.fork(a_method(object)) > end > --- > > But this still only used one CPU (and looks a bit ugly..). Is that some > limitation of ruby (v 1.8 specifically) or am I doing something wrong? > > Cheers, > > Marc > -- > Posted via http://www.ruby-forum.com/. > > -- Kind Regards, Rajinder Yadav http://DevMentor.org Do Good! - Share Freely, Enrich and Empower people to Transform their lives
From: Glen Holcomb on 29 Oct 2009 13:48 On Thu, Oct 29, 2009 at 8:56 AM, Marc Hoeppner <marc.hoeppner(a)molbio.su.se>wrote: > Hi, > > I've been reading around a bit but couldn't find a solution that worked, > so here goes: > > I am running ruby 1.8 and want to make full use of a quad core CPU > (64bit, Ubuntu) in a task that lends itself to multithreading/multicore > use. > > It's basically an array of objects that are each use in a fairly CPU > intensive job, so I figured I could have 4 of them run at the same time > , one on each CPU. > > BUT... > > The only reasonably understandably suggestion looked something like: > > ---- > threads = 4 > my_array = [something_here] > > threads.times do > Process.fork(a_method(my_array.shift)) > end > > my_array.each do |object| > Process.wait(0) > Process.fork(a_method(object)) > end > --- > > But this still only used one CPU (and looks a bit ugly..). Is that some > limitation of ruby (v 1.8 specifically) or am I doing something wrong? > > Cheers, > > Marc > -- > Posted via http://www.ruby-forum.com/. > > You are going to want Ruby 1.9 for this. In 1.8 threads are "green", basically they only exists as threads inside the VM so you still only hit one core and any blocking system I/O will block all of your threads. -- "Hey brother Christian with your high and mighty errand, Your actions speak so loud, I cant hear a word youre saying." -Greg Graffin (Bad Religion)
From: Peter Booth on 29 Oct 2009 15:55 Marc, How long lived is each of these tasks? Are we talking seconds or weeks? Is there a user-facing aspect to this or is throughput the variable that you're wanting to optimize? When you say "fairly CPU intensive", doe sthis mean that when one of these tasks runs you see (from sar/mpstat) that one of your CPUs is pinned? Peter On Oct 29, 2009, at 10:56 AM, Marc Hoeppner wrote: > Hi, > > I've been reading around a bit but couldn't find a solution that > worked, > so here goes: > > I am running ruby 1.8 and want to make full use of a quad core CPU > (64bit, Ubuntu) in a task that lends itself to multithreading/ > multicore > use. > > It's basically an array of objects that are each use in a fairly CPU > intensive job, so I figured I could have 4 of them run at the same > time > , one on each CPU. > > BUT... > > The only reasonably understandably suggestion looked something like: > > ---- > threads = 4 > my_array = [something_here] > > threads.times do > Process.fork(a_method(my_array.shift)) > end > > my_array.each do |object| > Process.wait(0) > Process.fork(a_method(object)) > end > --- > > But this still only used one CPU (and looks a bit ugly..). Is that > some > limitation of ruby (v 1.8 specifically) or am I doing something wrong? > > Cheers, > > Marc > -- > Posted via http://www.ruby-forum.com/. >
From: Tony Arcieri on 29 Oct 2009 16:04 [Note: parts of this message were removed to make it a legal post.] On Thu, Oct 29, 2009 at 11:48 AM, Glen Holcomb <damnbigman(a)gmail.com> wrote: > You are going to want Ruby 1.9 for this. In 1.8 threads are "green", > basically they only exists as threads inside the VM so you still only hit > one core and any blocking system I/O will block all of your threads. > Ruby 1.9 isn't going to help you when using threads to distribute computation across CPU cores. The Global VM Lock ensures that simultaneous computation is still limited to one core. JRuby, on the other hand, does not have this limitation. On MRI/1.9 I would recommend using multiple processes. -- Tony Arcieri Medioh/Nagravision
|
Next
|
Last
Pages: 1 2 3 4 Prev: Open an explorer window Next: Thread error "undefined method `keys' for nil:NilClass" |