[BUG][POSSIBLE FIXES]
Tim Olson
tim at jumpnet.com
Tue Oct 5 14:33:37 UTC 1999
Andres Valloud wrote:
> I also remember that for some floats, if you sent #roundTo: 0.01, the
> answer was
>
> blablabla.xy9999999999999999999999999999999999999999999999...
It depends upon what you mean by "the answer was".
0.01 cannot be represented exactly in floating point, so the result of
roundTo: 0.01 is inexact. In the 4 float operations performed during
the roundTo: (multiply, truncate, add 0.5, divide), the maximum rounding
error induced is around 2 ULPs (Units of Least Precision).
The Float>>printOn: method is biased towards printing the shortest
representation of the actual number, and will ignore up to 3 ULPs of
difference between the actual number and the one that is printed. So if
you look at the printed representation of a number that has been rounded
to 0.01, you should always get at most 2 digits after the decimal point
(even though a much longer printed representation may be slightly closer
to the actual value).
To see the exact printed representation, you can use
Float>>absPrintExactlyOn:base: , which will print the most accurate
representation possible.
For example:
n := 0.82788887379127 roundTo: 0.01.
n printString. "0.83"
String streamContents: [:strm | n absPrintExactlyOn: strm base: 10].
"0.8300000000000001"
-- Tim Olson
More information about the Squeak-dev
mailing list
|