[Vm-dev] Reproducible VM crash on Win32 with callbacks

[Guillermo Polito]

Hi all,

With Pablo we have been tracking a bug on win32 that produces a
segmentation fault on callback return. We can reproduce it 100% certainly
when running the Alien qsort example both in latest pharo and squeak
versions.

After some debugging, it would seem that the thunkEntry function is
over-optimized in 32 bits, corrupting the (C) stack. This was particularly
boring because compiling the VM in debug mode was taking the bug away
:-). We have cornered the bug and checked that callbacks do work ok if we
disable optimizations just for the thunkEntry function like this:

long
__attribute__((optimize("O0"))) thunkEntry(void *thunkp, sqIntptr_t *stackp)

The thing is that latest mingw which we use for compiling the windows VM
even in travis, now comes with gcc 7.4.0 which has a lot more of
optimizations than before. Just having O1 also produces the same error.

We have tried disabling some particular optimizations like
fno-combine-stack-adjustments but with no result so far.

The strange thing is that other callbacks like the ones coming from libgit
work ok.

Has somebody taken a look into this too?
How would you suggest that we move on with this?
>From our side, we think that using a pragma to disable optimizations for
thunkEntry in the case of win32 looks okeyish at least to make the bug go
away.

Cheers,
Guille & Pablo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/vm-dev/attachments/20190115/c0759cb3/attachment.html>