Numerics question: reading floating point constants
Andres Valloud
andresv at ntct1.com
Sun Apr 9 16:07:38 UTC 2006
Hello Andreas,
Saturday, April 8, 2006, 8:12:22 PM, you wrote:
AR> Notice how the last hex digits are different.
Floating point numbers work in base 2, while human code works in base
10. That factor of 5 cannot be put into a floating point number
accurately because it's coprime with 2. Therefore, things like 1/5
cannot be represented with base 2 floating point numbers. As such,
this is probably what you are running into.
It would work if the constants were expressed as
bits * 2^exponent
Then it would be fine. But
digits * 10^exponent
will typically cause approximation errors.
--
Best regards,
Andres mailto:andresv at ntct1.com
More information about the Squeak-dev
mailing list
|