Prev: How to convert a float into time
Next: Getting output of program with some DRb for good measure
From: Brian Candler on 7 Apr 2010 10:45
Nathan Macinnes wrote:
> Thanks for the clarification... My application is network based, and
> some operations take several seconds. generate_list_of_objects takes
> anything between 2 and 15 seconds, so I wouldn't like to have my whole
> program on hold while that's happening. Each test process takes a few
> seconds too, so if the object is no longer on the list, it isn't a
> particularly bad thing if test tries to run, but it'll take up a lot
> of time unnecessarily.

Let's say generate_list_objects takes 15 seconds, and the test suite
(iterating the objects) takes 60 seconds. Do you really want
generate_list_objects to run 4 times uselessly while the test suite runs
the original set of tests?

Similarly, if the test suite takes 15 seconds and generate_list_objects
takes 60 seconds, do you want to run the test suite 4 times identically
with the first set of objects?

If not, but rather you want to run the test suite exactly once for each
set of data, then you should synchronize them. How about this
(untested):

require 'thread'
cmd = Queue.new
res = Queue.new
tester = Thread.new do
while myHash = cmd.pop
myHash.each { ... }
res.push [myHash, :ok]
end
end

q.push MyClass.generate_list_of_objects
while Time.now < time_to_finish
q.push MyClass.generate_list_of_objects
res.pop
end
res.pop
q.push nil
tester.wait
--
Posted via http://www.ruby-forum.com/.

From: Brian Candler on 7 Apr 2010 10:46
Replace 'q' with 'cmd' in that of course.
--
Posted via http://www.ruby-forum.com/.

From: Nathan on 7 Apr 2010 10:53
> Do you really want generate_list_objects to run 4 times uselessly while
> the test suite runs the original set of tests?

I think you've misunderstood... sorry if I wasn't clear. Perhaps
naming the method test was a bad idea. I don't mean test as in unit
test or whatever, I mean something completely different. If an object
is no longer on the list returned by generate_list_of_objects, that
means it has become unavailable, and running test won't accomplish
anything meaningful. That's why I'm interested in keeping the list as
up-to-date as possible.

On Apr 7, 3:45 pm, Brian Candler <b.cand...(a)pobox.com> wrote:
> Nathan Macinnes wrote:
> > Thanks for the clarification... My application is network based, and
> > some operations take several seconds. generate_list_of_objects takes
> > anything between 2 and 15 seconds, so I wouldn't like to have my whole
> > program on hold while that's happening. Each test process takes a few
> > seconds too, so if the object is no longer on the list, it isn't a
> > particularly bad thing if test tries to run, but it'll take up a lot
> > of time unnecessarily.
>
> Let's say generate_list_objects takes 15 seconds, and the test suite
> (iterating the objects) takes 60 seconds. Do you really want
> generate_list_objects to run 4 times uselessly while the test suite runs
> the original set of tests?
>
> Similarly, if the test suite takes 15 seconds and generate_list_objects
> takes 60 seconds, do you want to run the test suite 4 times identically
> with the first set of objects?
>
> If not, but rather you want to run the test suite exactly once for each
> set of data, then you should synchronize them. How about this
> (untested):
>
> require 'thread'
> cmd = Queue.new
> res = Queue.new
> tester = Thread.new do
>   while myHash = cmd.pop
>     myHash.each { ... }
>     res.push [myHash, :ok]
>   end
> end
>
> q.push MyClass.generate_list_of_objects
> while Time.now < time_to_finish
>   q.push MyClass.generate_list_of_objects
>   res.pop
> end
> res.pop
> q.push nil
> tester.wait
> --
> Posted viahttp://www.ruby-forum.com/.

From: Nathan on 7 Apr 2010 10:57
Dan, many thanks for this. Method 2 looks like it'll suite me
perfectly. As I've explained in reply to Brian's post, I don't want to
test objects which are no longer on the most up-to-date list, and, if
I've understood correctly, method 1 would still do that, but method 2
wouldn't. So that looks like the solution.
Thanks,
Nathan

On Apr 7, 3:32 pm, "Dan Drew" <d...(a)brilliantsoftware.ca> wrote:
> Depending on how memory efficient you want to be you could also try.
>
> 1) Memory hog method... this will have potentially two copies of your data at a given time
>
> Thread 1:
>     lock_shared   # using whatever mechanism you want such as a mutex
>     testHash = sharedHash
>     unlock_shared
>     # iterate and test
>
> Thread 2
>     newHash = generate_hash()
>     lock_shared
>     sharedHash = newHash
>     unlock_shared
>
> 2) Memory efficient but more contention
>
> Thread 1
>     lock_shared
>     test_keys = sharedHash.keys
>     unlock_shared
>     test_keys.each do |k|
>         lock_shared
>         v = sharedHash[k]
>         unlock_shared
>         test(v) if v
>     end
>
> Thread 2
>     # for each item as it's loaded from the network
>     generate_index do |k,v|          
>         next if sharedHash[k] == v  # Optional to avoid unnecessary contention
>         lock_shared
>         if v
>             sharedHash[k] = v
>         else
>             sharedHash.delete(k)
>         end
>         unlock_shared
>     end
>
> Dan
>
> From: Matthew K. Williams
> Sent: Wednesday, April 07, 2010 10:02 AM
> To: ruby-talk ML
> Subject: Re: modifying a Hash in one process when .each is running in another
>
> On Wed, 7 Apr 2010, Nathan wrote:
> > Thanks for the clarification... My application is network based, and
> > some operations take several seconds. generate_list_of_objects takes
> > anything between 2 and 15 seconds, so I wouldn't like to have my whole
> > program on hold while that's happening. Each test process takes a few
> > seconds too, so if the object is no longer on the list, it isn't a
> > particularly bad thing if test tries to run, but it'll take up a lot
> > of time unnecessarily.
>
> > I'm trying to work out if there's another way of doing it then.
> > Perhaps I'll modify the test method so that it knows if the current
> > object is on the list, and only runs the test if it is.
> > Nathan
>
> Might a messaging queue work better?  That way you don't have the
> concurrency issues, as well as the issues with changing a hash in the
> middle of iteration.
>
> Matt

From: Nathan on 7 Apr 2010 10:59
Sorry, that should read "it's source has become unavailable".
Obviously the object itself is still there!!!

On Apr 7, 3:53 pm, Nathan <njmacin...(a)gmail.com> wrote:
> > Do you really want generate_list_objects to run 4 times uselessly while
> > the test suite runs the original set of tests?
>
> I think you've misunderstood... sorry if I wasn't clear. Perhaps
> naming the method test was a bad idea. I don't mean test as in unit
> test or whatever, I mean something completely different. If an object
> is no longer on the list returned by generate_list_of_objects, that
> means it has become unavailable, and running test won't accomplish
> anything meaningful. That's why I'm interested in keeping the list as
> up-to-date as possible.
>
> On Apr 7, 3:45 pm, Brian Candler <b.cand...(a)pobox.com> wrote:
>
>
>
> > Nathan Macinnes wrote:
> > > Thanks for the clarification... My application is network based, and
> > > some operations take several seconds. generate_list_of_objects takes
> > > anything between 2 and 15 seconds, so I wouldn't like to have my whole
> > > program on hold while that's happening. Each test process takes a few
> > > seconds too, so if the object is no longer on the list, it isn't a
> > > particularly bad thing if test tries to run, but it'll take up a lot
> > > of time unnecessarily.
>
> > Let's say generate_list_objects takes 15 seconds, and the test suite
> > (iterating the objects) takes 60 seconds. Do you really want
> > generate_list_objects to run 4 times uselessly while the test suite runs
> > the original set of tests?
>
> > Similarly, if the test suite takes 15 seconds and generate_list_objects
> > takes 60 seconds, do you want to run the test suite 4 times identically
> > with the first set of objects?
>
> > If not, but rather you want to run the test suite exactly once for each
> > set of data, then you should synchronize them. How about this
> > (untested):
>
> > require 'thread'
> > cmd = Queue.new
> > res = Queue.new
> > tester = Thread.new do
> >   while myHash = cmd.pop
> >     myHash.each { ... }
> >     res.push [myHash, :ok]
> >   end
> > end
>
> > q.push MyClass.generate_list_of_objects
> > while Time.now < time_to_finish
> >   q.push MyClass.generate_list_of_objects
> >   res.pop
> > end
> > res.pop
> > q.push nil
> > tester.wait
> > --
> > Posted viahttp://www.ruby-forum.com/.

First  |  Prev  |  Next  |  Last
Pages: 1 2 3
Prev: How to convert a float into time
Next: Getting output of program with some DRb for good measure