Prev: Correction needed in Copy File code
Next: XL app wants focus?
From: Eric on 17 Jun 2010 10:19
Hi,

In a data table I have records numbered from 801 to 809.
I also use these numbers as row number then 8.01 to 8.09.

To do so I just devide the record number by 100 and put it in a variable of
type decimal (also tried double and single).

At one point I compare this value with a number to do an evaluation:

itemnr = CDec(rw("itemnr")) / 100

"If itemnr = 8.03 Or itemnr = 8.04 Or itemnr = 8.06 Or itemnr = 8.07 Then"

Well it should look like that, but it actually showing:
" If itemnr = 8.0299999999999994 Or itemnr = 8.0399999999999991 Or itemnr =
8.0600000000000005 Or itemnr = 8.0700000000000003 Then"

How can I get rid of the extra decimals?
I don't need them and it's confusing.

rg,
Eric




From: J.B. Moreno on 17 Jun 2010 11:41
Eric <Eric(a)discussions.microsoft.com> wrote:

> Hi,
>
> In a data table I have records numbered from 801 to 809.
> I also use these numbers as row number then 8.01 to 8.09.
>
> To do so I just devide the record number by 100 and put it in a variable of
> type decimal (also tried double and single).
>
> At one point I compare this value with a number to do an evaluation:
>
> itemnr = CDec(rw("itemnr")) / 100
>
> "If itemnr = 8.03 Or itemnr = 8.04 Or itemnr = 8.06 Or itemnr = 8.07 Then"
>
> Well it should look like that, but it actually showing:
> " If itemnr = 8.0299999999999994 Or itemnr = 8.0399999999999991 Or itemnr =
> 8.0600000000000005 Or itemnr = 8.0700000000000003 Then"

Fractional numbers are always approximate --- if you're going to
proceed with this, you're going to have to deal with the rounding
issue.

I don't understand what you're doing well enough to say what you should
do, but I'd reconsider it if I was you....

--
J.B. Moreno
From: Armin Zingler on 17 Jun 2010 10:52
Am 17.06.2010 16:19, schrieb Eric:
> Hi,
>
> In a data table I have records numbered from 801 to 809.
> I also use these numbers as row number then 8.01 to 8.09.
>
> To do so I just devide the record number by 100 and put it in a variable of
> type decimal (also tried double and single).
>
> At one point I compare this value with a number to do an evaluation:
>
> itemnr = CDec(rw("itemnr")) / 100
>
> "If itemnr = 8.03 Or itemnr = 8.04 Or itemnr = 8.06 Or itemnr = 8.07 Then"
>
> Well it should look like that, but it actually showing:
> " If itemnr = 8.0299999999999994 Or itemnr = 8.0399999999999991 Or itemnr =
> 8.0600000000000005 Or itemnr = 8.0700000000000003 Then"
>
> How can I get rid of the extra decimals?
> I don't need them and it's confusing.

If 'itemnr' is Decimal or Double, I can't repro the problem:

Dim value As Decimal = 803
Dim itemnr As Double 'As Decimal: same behavior

itemnr = value / 100

If itemnr = 8.03 Then
MsgBox("equal") 'always shown
End If


--
Armin
From: OmegaSquared on 17 Jun 2010 13:19
Hello, Eric,

The numbers 8.xx are being interpreted as floating point values. Appending
a "D" to them should correct the problem that you are seeing. That is, try:

"If itemnr = 8.03D Or itemnr = 8.04D Or ...

It is very important to have a clear understanding of the different number
types that can be used and what their limitations are. Floating point values
particularly seem to cause some programmers much confusion. See:

http://docs.sun.com/source/806-3568/ncg_goldberg.html

for a good detailed explanation.

Cheers,
Randy

 | 
Pages: 1
Prev: Correction needed in Copy File code
Next: XL app wants focus?