[Vm-dev] [OpenSmalltalk/opensmalltalk-vm] Mixed SmallInteger/Float comparison is inexact in Spur64 when jitted (#417)

[Nicolas Cellier]

Hi David,
sort of... 

We know that `smallInt asFloat` may be inexact.
We could test whether it is exact or not with `smallInt asFloat asInteger = smallInt` like what is done in Squeak `SmallInteger>>#isAnExactFloat` check, but that's not what we do here.
What we do is to check if ever that rounding error could change the result of comparison. If it could not, then the rounding error is innocuous and we can proceed with float comparison.

So more exactly, the algorithm is:
- check if there is a possible ambiguity
- if yes, use exact comparison (the usual image side `smallInt = smallFloat asTrueFraction`)
- if no, use inexact but innocuous comparison (the simple `smallInt asFloat = smallFloat`)
  (strictly speaking, the comparison is exact, only the operands may not)

If `smallInt asFloat > smallFloat` we know for sure that `smallInt > smallFloat`.
If `smallInt asFloat < smallFloat` we know for sure that `smallInt < smallFloat`.
The only case where we could have ambiguity is when `smallInt asFloat = smallFloat`.

But in this case, we know that smallFloat value is integer. Indeed, either asFloat is exact, and `smallInt = smallFloat`, or inexact, but that means that `smallInt > (2 raisedTo: Float precision)` and then Float has not enough precision to have a fraction part. Thus `smallFloat asTrueFraction = smallFloat asInteger` in this ambiguous case, a nice thing for the VM to not deal with Fraction!

A potential remaining problem could be that smallFloat asInteger may be a LargeInteger, for example `SmallInteger maxVal asInteger > SmallInteger maxVal` in 64 bits image.
But we know that:
`SmallInteger minVal <= smallFloat and: [smallFloat <= SmallInteger maxVal nextPowerOfTwo]`
Thus in the VM, we are safe, SmallInteger span only 61 bits and we have 64 bits registers.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/417#issuecomment-523326042
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/vm-dev/attachments/20190821/0123ddf8/attachment.html>