But adding two extra digits of precision gives -937566.236469986849
in [C++]
|
From: Rick.Regan on 26 Jun 2010 23:57 When you round to 11 decimal places, sure, you'll get -937566.23646998685. But if you round to 12, you'll get -937566.236469986849. (Beware: You can see the latter on gcc, which prints all the digits -- but Visual C++ just adds 0s with each additional place: 85, 850, 8500, ...). To understand what's happening you have to know the value in the double; it is exactly -937566.236469986848533153533935546875. To 16 places it rounds to -937566.2364699868. Andrew wrote: I am not convinced. 26-Jun-10 I am not convinced. See my sample program in this thread that uses the value -937566.2364699869. When GCC takes that string, converts it to a double, then converts the double back to a string, it gives -937566.2364699868. Adding an extra digit of precision gives -937566.23646998685. IFAICS this means it is doing the rounding incorrectly. The real program (whose source code I cannot reproduce here) was written very quickly under quite a bit of pressure (and not by me BTW). There are a few corner cases where the memory demands are huge which is why the input numbers are converted from strings to doubles. My approach would have been to keep them as strings then there would be no conversion/precision problres. But this would have forced the program to solve the lack of memory problem. This can be done with various ways and means but would have made the program much more complex just to cover those few corner cases. We do not know yet how often these corner cases will come up in practice. If it proves to be a bigger problem than we thought then my suggestion is to build the program in 64 bit mode. This would allow much more memory to be addressed and would keep the program relatively simple. Regards, Andrew Marlow Previous Posts In This Thread: Submitted via EggHeadCafe - Software Developer Portal of Choice Composite UI Pattern And Enterprise Settings http://www.eggheadcafe.com/tutorials/aspnet/14dd2b7f-9da4-4a45-bc93-ce5fdba5c5ee/composite-ui-pattern-and-enterprise-settings.aspx -- [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ]
|
Pages: 1 Prev: I think GCC gets it right Next: Use of double type in float subtraction vs precision |