[squeak-dev] Float rounding issue

Levente Uzonyi leves at elte.hu
Mon May 7 04:20:33 UTC 2012


On Sun, 6 May 2012, Chris Muller wrote:

> ( 0.9580000000000001 roundTo: 0.0001) = 0.9580   "false"

Still didn't look at the code, but 0.0001 is also a Float and it doesn't 
have an exact representation in base 2. The following works as expected:
(0.9580000000000001 roundTo: 0.0001s4) "0.9580s4"
(0.9580000000000001 roundTo: 1/10000) "(479/500)"


Levente

>
>
>
> On Sun, May 6, 2012 at 8:55 PM, Levente Uzonyi <leves at elte.hu> wrote:
>> On Sun, 6 May 2012, Chris Muller wrote:
>>
>>> With the latest trunk updates, I'm having trouble rounding to my
>>> desired precision:
>>>
>>>    0.9580000000000001 roundTo: 0.0001    " 0.9580000000000001"
>>>
>>> In Squeak 4.3 I get what I expect:
>>>
>>>    0.9580000000000001 roundTo: 0.0001   " 0.958"
>>>
>>>
>>
>> I'm pretty sure it's just a difference in printing. 0.958 is unlikely to
>> have an exact base 2 representation.
>>
>>
>> Levente
>>
>
>


More information about the Squeak-dev mailing list