[BUG][POSSIBLE FIXES]

Tim Olson tim at jumpnet.com
Tue Oct 5 14:33:37 UTC 1999


Andres Valloud wrote:
> I also remember that for some floats, if you sent #roundTo: 0.01, the
> answer was
> 
> blablabla.xy9999999999999999999999999999999999999999999999...


It depends upon what you mean by "the answer was".

0.01 cannot be represented exactly in floating point, so the result of
roundTo: 0.01 is inexact.  In the 4 float operations performed during
the roundTo: (multiply, truncate, add 0.5, divide), the maximum rounding
error induced is around 2 ULPs (Units of Least Precision).

The Float>>printOn: method is biased towards printing the shortest
representation of the actual number, and will ignore up to 3 ULPs of
difference between the actual number and the one that is printed.  So if
you look at the printed representation of a number that has been rounded
to 0.01, you should always get at most 2 digits after the decimal point
(even though a much longer printed representation may be slightly closer
to the actual value).

To see the exact printed representation, you can use
Float>>absPrintExactlyOn:base: , which will print the most accurate
representation possible.

For example:

n := 0.82788887379127 roundTo: 0.01.

n printString.		"0.83"

String streamContents: [:strm | n absPrintExactlyOn: strm base: 10].

	"0.8300000000000001"


	-- Tim Olson





More information about the Squeak-dev mailing list