Prev: 1D to 3D coordinate translation. (testing hardware precision for 3d textures and inprecise 3d texture coordinates)
Next: Intel documentation mistake for single floating point precision ! (Nope oops lol)
From: Skybuck Flying on 2 Feb 2010 10:03
Intel document:

"
The IA-32 Intel Architecture Software Developer's Manual, Volume 1, Basic
Architecture, Document Number 253665 rev 19.pdf"

Contains a serious mistake on page 4-5:

"
Table 4-2. Length, Precision, and Range of Floating-Point Data Types
"

It reads:

"
Single precision: 24 bits
"

24 bits would mean:
2^24 = 16777216

In reality it seems limited to:
23 bits which means:
2^24 = 8388608

Following conceptual code shows the problem:
single = 8000 * 8000;
single = single - 1;
display single; // incorrect result

single = -8000 * 8000;
single = single - 1;
display single; // incorrect result

single = 2500 * 2500;
single = single - 1;
display single; // correct result

single = -2500 * 2500;
single = single - 1;
display single; // correct result

Bye,
Skybuck.


 | 
Pages: 1
Prev: 1D to 3D coordinate translation. (testing hardware precision for 3d textures and inprecise 3d texture coordinates)
Next: Intel documentation mistake for single floating point precision ! (Nope oops lol)