Hey, what's the URL to that IEEE floating-point number page?
in [Visual Basic]
|
Prev: Windows 7 taskbar API problems
Next: Show a previous instance of app running in system tray(Added Question for Karl)
From: Jeff Johnson on 30 Apr 2010 14:59 You know, the one people post when someone doesn't understand why 22 / 10 yields 2.2000000000000000000001 instead of 2.2.
From: dpb on 30 Apr 2010 15:27 Jeff Johnson wrote: > You know, the one people post when someone doesn't understand why 22 / 10 > yields 2.2000000000000000000001 instead of 2.2. > > Goldberg, I presume you mean??? <http://docs.sun.com/source/806-3568/ncg_goldberg.html> --
From: Patrice on 30 Apr 2010 15:46 Hello, > You know, the one people post when someone doesn't understand why 22 / 10 > yields 2.2000000000000000000001 instead of 2.2. You could try : http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems The short version is that floating point values are coded in a way that favor range over precision. The decimal data type allows to have full control over the precision. -- Patrice
From: Jeff Johnson on 30 Apr 2010 16:27 "dpb" <none(a)non.net> wrote in message news:hrfb2o$ate$1(a)news.eternal-september.org... >> You know, the one people post when someone doesn't understand why 22 / 10 >> yields 2.2000000000000000000001 instead of 2.2. > Goldberg, I presume you mean??? > > <http://docs.sun.com/source/806-3568/ncg_goldberg.html> Urk. Is this the one Ken (I think it was Ken...maybe Rick?) used to post all the time? There's a ton of math in the one you linked to, and I think this other one was a bit more...approachable.
From: Rick Rothstein on 30 Apr 2010 17:14
> Urk. Is this the one Ken (I think it was Ken...maybe Rick?) used to post > all the time? These are the three links I used to post (the last one being the one already posted)... INFO: Visual Basic and Arithmetic Precision http://support.microsoft.com/default.aspx?scid=http://support.microsoft.com:80/support/kb/articles/Q279/7/55.ASP&NoWebContent=1 (Complete) Tutorial to Understand IEEE Floating-Point Errors http://support.microsoft.com/default.aspx?scid=http://support.microsoft.com:80/support/kb/articles/Q42/9/80.ASP&NoWebContent=1 What Every Computer Scientist Should Know About Floating Point http://docs.sun.com/source/806-3568/ncg_goldberg.html -- Rick (MVP - Excel) |