[Newbies] ScaledDecimal implementation (was: Number formatting printf/sprintf for Squeak?)

[Chris Cunningham]

On Wed, Feb 18, 2009 at 12:21 PM, Michael van der Gulik
<mikevdg at gmail.com>wrote:

>
> Could you give me an example? It isn't immediately intuitive to me what
> incorrect results might occur. Rounding errors maybe?
>
Well, I had to go back and figure out what they were ( I don't have the
exact examples that caused this issue in the first place).  However, if you
have an intermediate calculation that results in a fraction more precise
than what you need, and you make a scaledDecimal out of it, you can get
weird results.  The example:

33.33 + 33.33 + 33.33 + 33.33 = 133.32 "in real math"

x := 33.333s2.   "This prints as 33.33s2, giving the illusion that you have
it exactly as this amount"
x + x + x + x = 133.33s2  "Not accurate, depending on exactly what you are
trying to measure"

y := 33.333 asFixedDecimal: 2.  "This prints as 33.33"
y + y + y + y = 133.32  "that is, the fixed decimal of scale 2 truely
truncated the value, and the result is correct, depending on what you are
measuring"

So, if you need to keep the original precision and you are just want to
present it at a lower precision, ScaledDecimal is exactly what you want.
If, on the other hand, you want the number to be at exactly the precision
that you tell it it should be, then FixedDecimal is what you want.

-Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/beginners/attachments/20090218/abb15a81/attachment.htm