[squeak-dev] Float precision
David T. Lewis
lewis at mail.msen.com
Wed Dec 22 13:38:01 UTC 2010
On Wed, Dec 22, 2010 at 08:04:11AM +0100, Michael Haupt wrote:
> Hi,
>
> Am 22.12.2010 um 06:19 schrieb Markus Lampert <markuslampert at yahoo.com>:
> > Evaluating the following:
> >
> > 0.0 to: 10 by: 0.1 do: [ :v |
> > Transcript show: v printString; crlf]
> >
> > gives some interesting numbers for 3.6, 3.7, 3.8, 3.9 and everything above 5.5.
> > Is this expected given the rounding effects of Float?
>
> inaccuracies are always expected with floating-point numbers. If you want accurate results, you should use BigDecimal (IIRC) or SmallInteger and manual decimal point management.
>
> Best,
>
> Michael
Yes. And that would be ScaledDecimal (not BigDecimal).
This is a common source of confusion. Wikipedia gives a good overview
of the topic:
http://en.wikipedia.org/wiki/Floating_point
Dave
More information about the Squeak-dev
mailing list
|