[squeak-dev] Float rounding issue
Levente Uzonyi
leves at elte.hu
Mon May 7 04:20:33 UTC 2012
On Sun, 6 May 2012, Chris Muller wrote:
> ( 0.9580000000000001 roundTo: 0.0001) = 0.9580 "false"
Still didn't look at the code, but 0.0001 is also a Float and it doesn't
have an exact representation in base 2. The following works as expected:
(0.9580000000000001 roundTo: 0.0001s4) "0.9580s4"
(0.9580000000000001 roundTo: 1/10000) "(479/500)"
Levente
>
>
>
> On Sun, May 6, 2012 at 8:55 PM, Levente Uzonyi <leves at elte.hu> wrote:
>> On Sun, 6 May 2012, Chris Muller wrote:
>>
>>> With the latest trunk updates, I'm having trouble rounding to my
>>> desired precision:
>>>
>>> 0.9580000000000001 roundTo: 0.0001 " 0.9580000000000001"
>>>
>>> In Squeak 4.3 I get what I expect:
>>>
>>> 0.9580000000000001 roundTo: 0.0001 " 0.958"
>>>
>>>
>>
>> I'm pretty sure it's just a difference in printing. 0.958 is unlikely to
>> have an exact base 2 representation.
>>
>>
>> Levente
>>
>
>
More information about the Squeak-dev
mailing list
|