Eliot Miranda <eliot.miranda <at> gmail.com> writes:

> 
> 
> Hi Nicholas,
> 
>     the x87 FPU (embedded in all modern x86's and used by default to pass
floating point arguments in ABI calls) is a stack of only 8 registers.
> 
> Imagine what happens if you attempt an FFI call that parse, say, 4 of its
floating-point arguments before hitting an invalid parameter.  Likely the FFI
marshalling machinery will leave 4 items on the x87's stack and forget to clean
them up.
> 
> Imagine what happens if you call some broken code that doesn't leave the x87
register stack empty.
> 
> In either of these cases the next time you try and make a call and load items
onto the x87's stack it'll quite possibly overflow.
> 

Yes, that's exactly what happens in my test case (gdb output provided at
http://bugs.squeak.org/view.php?id=3929).
After the seventh call, there are 7 slots occupied on stack.
If I then attempt a comparison of the result with another float, I need two
slots, the stack overflows, pushes a qNan and the answer of comparison is always
false.

But this is not because of the Callee: it seems weel behaved, the only double
being on FPU stack on function return is the returned double value.
It is the FFI glue Caller itself which fails to unstack the result.

Changing line 51 of x86-sysv-asm.S solved this problem:

    fstpl ffiFloatReturnValue

But that's only a guess, I'm not sure if there are other implications.


> So.... the FFI marshalling machinery really needs to reset the FPU's stack
pointer to zero both before and after making a call.  I don't know if the FFI
does this but chances are it doesn't.
> 
> In my time VisualWorks suffered problems like this, including its machine code
floating point primitives not zeroing the x87 stack pointer on primitive fail.
 Now it does an fninit (the relevant instruction to intialize the x87 FPU) in
relevant places.
> 
> Cheers
> Eliot
> 

As suggested by my above tests, the fninit does not seem to be called in FFI.
Would you know how to force such an instruction in a portable way via VMMaker?

There must be some fninit called somewhere in the VM, because I see the stack is
reset after a few UI. That would be interesting to know if they are in the right
place indeed. Only you (and a few others) could check I'm afraid.

Thank you

Nicolas