Prev: ActionController::AbstractRequest.relative_url_root = "/rotspecs"
Next: Now Win a laptop computer from (EZ laptop) free of charge
From: Yotta Meter on 23 Feb 2010 14:27
One edit to the above:

When 'flunk' is replaced with true:
Finished in 0.482389 seconds.
12221 tests, 12221 assertions, 12221 failures, 0 errors

should read

When 'flunk' is replaced with true:
Finished in 0.482389 seconds.
12221 tests, 0 assertions, 0 failures, 0 errors

I got lazy and copied only the time from the previous value.
--
Posted via http://www.ruby-forum.com/.

From: Brian Candler on 23 Feb 2010 14:34
Yotta Meter wrote:
> The above would indicate that the problem is not with define_method, but
> rather the ability of Test::Unit to handle a large number of test cases.

A large number of *failing* test cases. What if you do

def flunk
end

or

def flunk
assert true
end

?

> I couldn't get define_method going in a class to compare, it's a private
> method in Method, and I really don't understand the Method rdoc example
> using send.

class Foo
define_method ...
end

should work fine. If you have an existing Class object you can also do

Foo.class_eval { define_method ... }

> Is there some sort of optimization in Test::Unit that I need to set?
> Seems like something is really wrong, where is all this time going?

Rescuing exceptions perhaps. I thought you said that you expected most
of your tests to pass, and only a few to fail?

--
Posted via http://www.ruby-forum.com/.

From: Walton Hoops on 23 Feb 2010 14:43
On 2/23/2010 12:20 PM, Yotta Meter wrote:
> I'm seeing some performance issues I can't seem to get around. Your
> solution for creating a method was great though. I'd try the last one,
> but I think I'm seeing a fundamental problem with Test::Unit.
>
> Using the following code as the base reference:
>
> #!/usr/bin/ruby
>
> require "test/unit"
>
> class TestImplArchFile < Test::Unit::TestCase
> (0..10).each do |i|
> (0..10).each do |j|
> (0..100).each do |k|
> define_method("test_i#{i}__j#{j}_k#{k}") do
> flunk
> end
> end
> end
> end
> end
>
>
> When 'flunk' is set to flunk
> Finished in 5.586734 seconds.
> 12221 tests, 12221 assertions, 12221 failures, 0 errors
>
> When 'flunk' is replaced with true:
> Finished in 0.482389 seconds.
> 12221 tests, 12221 assertions, 12221 failures, 0 errors
>
> If I manually generate all 12k tests as individual tests in a new file,
> with each one flunking:
> Finished in 5.631014 seconds.
> 12221 tests, 12221 assertions, 12221 failures, 0 errors
>
> The above would indicate that the problem is not with define_method, but
> rather the ability of Test::Unit to handle a large number of test cases.
>
> As a comparison, if I just print out results in a log file, I get the
> awesome result:
>
> Just using a class, no Unit::Test, not creating method, just print
> method name
> Finished in 0.102 seconds.
>
> I couldn't get define_method going in a class to compare, it's a private
> method in Method, and I really don't understand the Method rdoc example
> using send.
>
> Is there some sort of optimization in Test::Unit that I need to set?
> Seems like something is really wrong, where is all this time going?
>
Nothings wrong, exceptions are just slow. This is actually true of most
programming languages. 'flunk' fails by generating an exception and
Test::Unit proceeds to rescue that exception. For some more information
on just how slow exceptions are, check out:
http://www.simonecarletti.com/blog/2010/01/how-slow-are-ruby-exceptions/

This is why and exception should almost never be the expected outcome
of running any piece of code. Likewise, failure should never be the
expected
outcome of 12221 tests. If you have 12221 tests failing, you have much
bigger
problems than a 5 second run-time on your test suite.

From: Yotta Meter on 23 Feb 2010 15:20
Yes, the all passing test ran in the 0.48 sec posted above. For the most
part, I expect all to be passing, but I wanted to understand the range
of values.

The problem is I have more than 10k tests. I was just using that as a
relative benchmark.

Also, the 0.48 sec will go up when the comparison happens with complex
objects. This is sort of best case.

In response to Walton's statement, the 5 seconds isn't the focus, it's
really the half second. What it's saying is that for 10k basic level
tests with all passing, it takes half a second. In our system we
integrate from a number of groups, and I was hoping to use Test::Unit as
a framework to verify all object relationships. It's not just the
Exceptions, there is something in Test::Unit that is taking more time
than it should, as the half second doesn't come close to the tenth of a
second of the class implementation. Both of those examples do not have
exceptions. I threw in the exceptions just to see what the worst case
was. In reality, we will have tests in the millions, so I'm better off
using a Class than Test::Unit, or creating my own Test::Unit.

Richard, thanks for the reference on Dust, I'll be sure to check it out.
My concern is that as a wrapper around Test::Unit, I'll end up with the
same performance issue as before.

That all being said, we are dealing with large object association that
are unique to our system. When we simulate our ASIC with vendor software
it takes approximately 100G of memory on a 64-bit system. I'm attempting
to build front end checks that would verify structure before the large
simulation occurs.

Thanks for all the help guys.
--
Posted via http://www.ruby-forum.com/.

From: Brian Candler on 23 Feb 2010 16:24
Yotta Meter wrote:
> Yes, the all passing test ran in the 0.48 sec posted above. For the most
> part, I expect all to be passing, but I wanted to understand the range
> of values.
>
> The problem is I have more than 10k tests. I was just using that as a
> relative benchmark.
>
> Also, the 0.48 sec will go up when the comparison happens with complex
> objects. This is sort of best case.

Well, Test::Unit is doing a degree of book-keeping for you - counting
the passing tests as well as assertions, writing a dot to the screen for
each test passed - and I'm also not sure how efficient it is to have
>10k methods in one class.

> In reality, we will have tests in the millions, so I'm better off
> using a Class than Test::Unit, or creating my own Test::Unit.

You could be right. Test::Unit was probably not designed for the case
where you have millions of tests and each test is so fast that the
overhead of Test::Unit itself is large.

You might want to try this first:

require "test/unit"

class TestImplArchFile < Test::Unit::TestCase
def flunk
assert true
end

def no_stop
yield
rescue Test::Unit::AssertionFailedError => e
add_failure(e.message, e.backtrace)
rescue Exception
raise if PASSTHROUGH_EXCEPTIONS.include? $!.class
add_error($!)
end

def test_all
(0..10).each do |i|
(0..10).each do |j|
(0..100).each do |k|
no_stop do
flunk
end
end
end
end
end
end

It runs about 4 times faster on my PC, possibly due to the tests not
being dispatched as separate methods (you'll need to use the profiler if
you want to understand exactly why). If it's not fast enough, then
having your own book-keeping is probably the right way to go.
--
Posted via http://www.ruby-forum.com/.

First  |  Prev  |  Next  |  Last
Pages: 1 2 3
Prev: ActionController::AbstractRequest.relative_url_root = "/rotspecs"
Next: Now Win a laptop computer from (EZ laptop) free of charge