tim Rowledge wrote
> As a side-effect of running the SpaceTally code I had to look at
> implementations of #roundTo: because the output was bizarrely formatted. 
> 
> It turns out that for several of the percentage values calculated by -
> percent := s spaceForInstances*100.0/totalInstSpace roundTo: 0.1.
> - we get decidedly not numbers that match what we probably think we should
> get. For example 
> 28846801 *100.0 /  53172599 ->  54.25125260474855
> BUT  
> 54.25125260474855 roundTo: 0.1 -> 54.300000000000004
> What? That's not even correct, let alone rounded to the requested
> precision.
> 
> Who is a numerics aficionado?
> 
> tim
> --
> tim Rowledge; 

> tim@

> ; http://www.rowledge.org/tim
> Managing programmers is like herding cats.

roundTo: rounds to a quantum so:

100 roundTo: 8 -> 104
100 roundTo: 0.8 -> 100
100 roundTo: (Float pi) -> 100.53096491487338

For printing you want:
54.25125260474855 printShowingDecimalPlaces: 1 -> '54.3'



--
View this message in context: http://forum.world.st/The-joys-or-not-of-floating-point-numbers-tp4678288p4678289.html
Sent from the Squeak - Dev mailing list archive at Nabble.com.