Intel documentation mistake for single floating point precision !
|
Prev: Predicted critical block NUCA
Next: Intel documentation mistake for single floating point precision ! (Nope oops lol)
From: Skybuck Flying on 2 Feb 2010 10:03 Intel document: " The IA-32 Intel Architecture Software Developer's Manual, Volume 1, Basic Architecture, Document Number 253665 rev 19.pdf" Contains a serious mistake on page 4-5: " Table 4-2. Length, Precision, and Range of Floating-Point Data Types " It reads: " Single precision: 24 bits " 24 bits would mean: 2^24 = 16777216 In reality it seems limited to: 23 bits which means: 2^24 = 8388608 Following conceptual code shows the problem: single = 8000 * 8000; single = single - 1; display single; // incorrect result single = -8000 * 8000; single = single - 1; display single; // incorrect result single = 2500 * 2500; single = single - 1; display single; // correct result single = -2500 * 2500; single = single - 1; display single; // correct result Bye, Skybuck. |