Prev: read attached document from .eml file
Next: Open an explorer window
From: Aldric Giacomoni on 28 Oct 2009 13:57
I found this blog entry:
http://raveendran.wordpress.com/2009/06/29/ruby-big-decimal/
Which shows the following..

irb(main):011:0> require 'bigdecimal'
=> true
irb(main):012:0> x = BigDecimal("123.6") - BigDecimal("123")
=> #<BigDecimal:28c5b84,'0.6E0',4(16)>
irb(main):013:0> puts x.to_f
0.6
=> nil
irb(main):014:0> puts f=123.6 - 123
0.599999999999994
=> nil
irb(main):015:0> 123.6 - 123
=> 0.599999999999994
irb(main):016:0> 123.6 - 123.0
=> 0.599999999999994
irb(main):017:0>

That's a little strange.. Isn't it?
--
Posted via http://www.ruby-forum.com/.

From: Matthew K. Williams on 28 Oct 2009 14:02
On Thu, 29 Oct 2009, Aldric Giacomoni wrote:

> I found this blog entry:
> http://raveendran.wordpress.com/2009/06/29/ruby-big-decimal/
> Which shows the following..
>
> irb(main):011:0> require 'bigdecimal'
> => true
> irb(main):012:0> x = BigDecimal("123.6") - BigDecimal("123")
> => #<BigDecimal:28c5b84,'0.6E0',4(16)>
> irb(main):013:0> puts x.to_f
> 0.6
> => nil
> irb(main):014:0> puts f=123.6 - 123
> 0.599999999999994
> => nil
> irb(main):015:0> 123.6 - 123
> => 0.599999999999994
> irb(main):016:0> 123.6 - 123.0
> => 0.599999999999994
> irb(main):017:0>
>
> That's a little strange.. Isn't it?

No, it's not. Welcome to the wonderfully confusing world of floating
point math...

From: Aldric Giacomoni on 28 Oct 2009 14:07
Matthew K. Williams wrote:
>> irb(main):016:0> 123.6 - 123.0
>> => 0.599999999999994
>>
>> That's a little strange.. Isn't it?
>
> No, it's not. Welcome to the wonderfully confusing world of floating
> point math...

Oh, thanks. Can I have some pop-corn and an introductory pamphlet before
I bash my head against the wall? :)
--
Posted via http://www.ruby-forum.com/.

From: Matthew K. Williams on 28 Oct 2009 14:21
On Thu, 29 Oct 2009, Aldric Giacomoni wrote:

> Matthew K. Williams wrote:
>>> irb(main):016:0> 123.6 - 123.0
>>> => 0.599999999999994
>>>
>>> That's a little strange.. Isn't it?
>>
>> No, it's not. Welcome to the wonderfully confusing world of floating
>> point math...
>
> Oh, thanks. Can I have some pop-corn and an introductory pamphlet before
> I bash my head against the wall? :)

Pamphlet -> http://en.wikipedia.org/wiki/IEEE_754-2008

Popcorn, well, it's kinda hard to transmit over the wire. ;-)

As a rule of thumb, if you really care about the decimals, either use
BigDecimal or integers (and keep track of where the decimal should be --
this is common for $$$$). Unfortunately, this is not limited to ruby,
either -- C, Java, and a host of other languages all are subject.

Matt

From: Aldric Giacomoni on 28 Oct 2009 14:48
Matthew K. Williams wrote:
> On Thu, 29 Oct 2009, Aldric Giacomoni wrote:
>
>> I bash my head against the wall? :)
> Pamphlet -> http://en.wikipedia.org/wiki/IEEE_754-2008
>
> Popcorn, well, it's kinda hard to transmit over the wire. ;-)
>
> As a rule of thumb, if you really care about the decimals, either use
> BigDecimal or integers (and keep track of where the decimal should be --
> this is common for $$$$). Unfortunately, this is not limited to ruby,
> either -- C, Java, and a host of other languages all are subject.
>
> Matt

Thanks. Both my head and the wall are safe for now. As one may have
surmised, I don't deal with floating point values much, but this is
frustrating. I do see that there are a couple of people who wrote gems
on Rubyforge to handle this issue :)
--
Posted via http://www.ruby-forum.com/.

 |  Next  |  Last
Pages: 1 2 3 4 5
Prev: read attached document from .eml file
Next: Open an explorer window