Arithmetic error

Thread view:

Enable EMail Alerts

Start New Thread

Thread rating:

maxarbos - 09 Apr 2008 16:52 GMT

I appoligize that this has been discussed, but I didn't see any
examples as simple as this. Three user input values are simply
subtracted: 55.45 - 52.37 - 3.08 = 0 However, excel calculates to
0.000000000000005329070518200750. They are user input values and so
are not rounded, however, I checked and verified 0's out to 30 places.

The only reason I noticed this is becuase in a matrix of results from
various inputs, all of which should have been 0 and with Tools/Options/
View set to clear '0' cells, I've got this one cell showing up.

I assume this is some sort of binary conversion error?

Thanks,
Dave

Reply to this Message

Jim Rech - 09 Apr 2008 18:13 GMT

It is the effect of decimal to binary (IEEE 64 bit) to decimal conversion.
I'm not sure it is an error though, at least in the sense of a bug or a
mistake. It's the result of the inherent and expected imprecision in this
process.

Signature

Jim

|I appoligize that this has been discussed, but I didn't see any
| examples as simple as this. Three user input values are simply
[quoted text clipped - 10 lines]
| Thanks,
| Dave

Reply to this Message

WGD - 10 Apr 2008 00:22 GMT

Have experienced likewise. Is the best way to "fix" to use a Rounding
function? While a year+ ago, I believe that is what I did.

> It is the effect of decimal to binary (IEEE 64 bit) to decimal conversion.
> I'm not sure it is an error though, at least in the sense of a bug or a
[quoted text clipped - 15 lines]
> | Thanks,
> | Dave

Reply to this Message

Jim Rech - 10 Apr 2008 13:42 GMT

I use the Tools, Options, Calculation, Precision as Displayed option
religiously. Of course before applying it to an existing workbook you have
to be sure that all cells have the number format that you want them rounded
to.

Signature

Jim

| Have experienced likewise. Is the best way to "fix" to use a Rounding
| function? While a year+ ago, I believe that is what I did.
[quoted text clipped - 3 lines]
| > mistake. It's the result of the inherent and expected imprecision in this
| > process.

news:f1b10e74-8954-4f4f-8d0b-406894bc4168@a70g2000hsh.googlegroups.com...

| > |I appoligize that this has been discussed, but I didn't see any
| > | examples as simple as this. Three user input values are simply
[quoted text clipped - 10 lines]
| > | Thanks,
| > | Dave

Reply to this Message

Rate this thread: