[squeak-dev] truncatedTo: / roundTo: oddities with fractions
David T. Lewis
lewis at mail.msen.com
Wed Jul 31 12:24:08 UTC 2019
On Wed, Jul 31, 2019 at 11:53:38AM +0000, Rein, Patrick wrote:
> Hi everyone,
> a student recently encountered the following behavior:
> fraction := 5 / 9.
> (fraction* 100) truncateTo: 0.01. ???55.550000000000004"
> fraction:= 1/18.
> (fraction * 100) roundTo: 0.01. ???5.5600000000000005"
> This is not what I would expect from #truncateTo: or #roundTo: even when using Floats (especially roundTo:).
> Is this what we want or an open issue? I have not found any test covering the protocol.
The results are unexpected, but not wrong.
Why unexpected? When I look at the expression, I intuitively expect
it to perform decimal arithmetic, and it does not do that.
Why not wrong? 0.01 is a float, even though it may have been intended
as an exact decimal number by the writer. Mixing float values in any
arithmetic expression produces an inexact float result, as it should.
To get a result that is both correct and intuitively right, the expression
might better be written like this:
5 / 9 * 100 truncateTo: (1/100) ==> (1111/20)
More information about the Squeak-dev