[squeak-dev] Re: x86 linux VM FFI FPU stack overflow

Eliot Miranda eliot.miranda at gmail.com
Tue Jan 13 22:36:15 UTC 2009

Hi Nicholas,
    the x87 FPU (embedded in all modern x86's and used by default to pass
floating point arguments in ABI calls) is a stack of only 8 registers.

Imagine what happens if you attempt an FFI call that parse, say, 4 of its
floating-point arguments before hitting an invalid parameter.  Likely the
FFI marshalling machinery will leave 4 items on the x87's stack and forget
to clean them up.

Imagine what happens if you call some broken code that doesn't leave the x87
register stack empty.

In either of these cases the next time you try and make a call and load
items onto the x87's stack it'll quite possibly overflow.

So.... the FFI marshalling machinery really needs to reset the FPU's stack
pointer to zero both before and after making a call.  I don't know if the
FFI does this but chances are it doesn't.

In my time VisualWorks suffered problems like this, including its machine
code floating point primitives not zeroing the x87 stack pointer on
primitive fail.  Now it does an fninit (the relevant instruction to
intialize the x87 FPU) in relevant places.


On Tue, Jan 13, 2009 at 11:12 AM, nicolas cellier <ncellier at ifrance.com>wrote:

> nicolas cellier a écrit :
>> Apologies for the cryptic subject...
>> I have some news about http://bugs.squeak.org/view.php?id=3929
>> I detected with gdb that the example I provided triggers a FPU stack
>> overflow. When this occurs, a qNan is placed on FPU stack top.
>> Thus, though ffiFloatReturnValue == 0.0, testing in Smalltalk (self
>> myFFICall = 0.0) will result in 0.0==Nan due to stack overflow.
>> primitive which execute special send #= will incorrectly return false...
>> Don't tell me to not use = on Floats, this is not the problem, you can
>> replace the test with (self myFFICall < 1.0), you will get (0.0<qNan) ->
>> false all the same.
>> At this point, I see two possibilities:
>> - either I totally messed up with calling conventions
>> - or the FFI glue ffiCallAddressOf is broken
>> ($VM/Squeak-3.10-1/platforms/unix/plugins/SqueakFFIPrims/x86-sysv-asm)
>> I have no knowledge about IA-32 and don't wish to learn further, using gdb
>> at instruction level was already painfull enough for me.
>> Can somebody aware of x86 and calling convention check my guess?
>> Nicolas
> Any one that can provide light on this subject?
> Or just get a contact with Ian Piumarta?
> I feel a bit alone
> Nicolas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20090113/7ed6353b/attachment.htm

More information about the Squeak-dev mailing list