[squeak-dev] Float rounding issue

Darius Clarke socinian at gmail.com
Mon May 7 05:51:24 UTC 2012


I don't believe roundTo: is the method you're looking for.
I've been confused by this before.

roundTo: quantum
"Answer the nearest number that is a multiple of quantum."

^(self / quantum) rounded * quantum

Maybe try this:

roundedToFloat: quantum
 "Round to decimal places."
| aReciprocal |
aReciprocal := quantum reciprocal rounded.

^(self * aReciprocal) rounded / aReciprocal asFloat



On Sun, May 6, 2012 at 9:20 PM, Levente Uzonyi <leves at elte.hu> wrote:

> On Sun, 6 May 2012, Chris Muller wrote:
>
>  ( 0.9580000000000001 roundTo: 0.0001) = 0.9580   "false"
>>
>
> Still didn't look at the code, but 0.0001 is also a Float and it doesn't
> have an exact representation in base 2. The following works as expected:
> (0.9580000000000001 roundTo: 0.0001s4) "0.9580s4"
> (0.9580000000000001 roundTo: 1/10000) "(479/500)"
>
>
> Levente
>
>
>>
>>
>> On Sun, May 6, 2012 at 8:55 PM, Levente Uzonyi <leves at elte.hu> wrote:
>>
>>> On Sun, 6 May 2012, Chris Muller wrote:
>>>
>>>  With the latest trunk updates, I'm having trouble rounding to my
>>>> desired precision:
>>>>
>>>>    0.9580000000000001 roundTo: 0.0001    " 0.9580000000000001"
>>>>
>>>> In Squeak 4.3 I get what I expect:
>>>>
>>>>    0.9580000000000001 roundTo: 0.0001   " 0.958"
>>>>
>>>>
>>>>
>>> I'm pretty sure it's just a difference in printing. 0.958 is unlikely to
>>> have an exact base 2 representation.
>>>
>>>
>>> Levente
>>>
>>>
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20120506/2eaa122c/attachment.htm


More information about the Squeak-dev mailing list