[squeak-dev] Float rounding issue

Darius Clarke socinian at gmail.com
Tue May 8 07:04:27 UTC 2012


I guess the "general rule with floats" is "Never use = or equivalence to
compare them. Only use > or < or check within a tolerance".

- Darius

On Mon, May 7, 2012 at 8:16 AM, Chris Muller <ma.chris.m at gmail.com> wrote:

> Ok, thanks Darius (and Levente).  At first I thought we had a
> regression but it looks like we don't..   :)
>
>
> On Mon, May 7, 2012 at 12:51 AM, Darius Clarke <socinian at gmail.com> wrote:
> > I don't believe roundTo: is the method you're looking for.
> > I've been confused by this before.
> >
> > roundTo: quantum
> > "Answer the nearest number that is a multiple of quantum."
> >
> > ^(self / quantum) rounded * quantum
> >
> > Maybe try this:
> >
> > roundedToFloat: quantum
> > "Round to decimal places."
> > | aReciprocal |
> > aReciprocal := quantum reciprocal rounded.
> >
> > ^(self * aReciprocal) rounded / aReciprocal asFloat
> >
> >
> >
> > On Sun, May 6, 2012 at 9:20 PM, Levente Uzonyi <leves at elte.hu> wrote:
> >>
> >> On Sun, 6 May 2012, Chris Muller wrote:
> >>
> >>> ( 0.9580000000000001 roundTo: 0.0001) = 0.9580   "false"
> >>
> >>
> >> Still didn't look at the code, but 0.0001 is also a Float and it doesn't
> >> have an exact representation in base 2. The following works as expected:
> >> (0.9580000000000001 roundTo: 0.0001s4) "0.9580s4"
> >> (0.9580000000000001 roundTo: 1/10000) "(479/500)"
> >>
> >>
> >> Levente
> >>
> >>>
> >>>
> >>>
> >>> On Sun, May 6, 2012 at 8:55 PM, Levente Uzonyi <leves at elte.hu> wrote:
> >>>>
> >>>> On Sun, 6 May 2012, Chris Muller wrote:
> >>>>
> >>>>> With the latest trunk updates, I'm having trouble rounding to my
> >>>>> desired precision:
> >>>>>
> >>>>>    0.9580000000000001 roundTo: 0.0001    " 0.9580000000000001"
> >>>>>
> >>>>> In Squeak 4.3 I get what I expect:
> >>>>>
> >>>>>    0.9580000000000001 roundTo: 0.0001   " 0.958"
> >>>>>
> >>>>>
> >>>>
> >>>> I'm pretty sure it's just a difference in printing. 0.958 is unlikely
> to
> >>>> have an exact base 2 representation.
> >>>>
> >>>>
> >>>> Levente
> >>>>
> >>>
> >>
> >>
> >>
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20120508/8158d40b/attachment.htm


More information about the Squeak-dev mailing list