Numerics question: reading floating point constants

Andres Valloud andresv at ntct1.com
Sun Apr 9 16:07:38 UTC 2006


Hello Andreas,

Saturday, April 8, 2006, 8:12:22 PM, you wrote:

AR> Notice how the last hex digits are different.

Floating point numbers work in base 2, while human code works in base
10.  That factor of 5 cannot be put into a floating point number
accurately because it's coprime with 2.  Therefore, things like 1/5
cannot be represented with base 2 floating point numbers.  As such,
this is probably what you are running into.

It would work if the constants were expressed as

  bits * 2^exponent

Then it would be fine.  But

  digits * 10^exponent

will typically cause approximation errors.

-- 
Best regards,
 Andres                            mailto:andresv at ntct1.com




More information about the Squeak-dev mailing list