[Vm-dev] [OpenSmalltalk/opensmalltalk-vm] Squeak3D crash on 64bits VM (#468)

[Nicolas Cellier]

Here is lldb debug report on OSX:

```
Process 95068 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x321089be150)
    frame #0: 0x00000001003c58fd Squeak`alphaSourceBlendBits32 at BitBltPlugin.c:820:5
   817 sourceWord = long32At(srcIndex);
   818 srcAlpha = ((usqInt) sourceWord) >> 24;
   819 if (srcAlpha == 0xFF) {
-> 820 long32Atput(dstIndex, sourceWord);
   821 srcIndex += 4;

```
we have :
```
(lldb) p/x dstIndex
(sqInt) $9 = 0x00000321089be150
(lldb) p/x destBits
(sqInt) $14 = 0x00000001089be470

```
Wow, `dstIndex` is very far from `destBits`... because:
```
(lldb) p/x dy
(sqInt) $13 = 0x00000000ffffffff
(lldb) p/x dstY
(sqInt) $15 = 0x00000000ffffffff

```
Hmm is that intended to be so large? or just -1?
Is it a missing sign extension?
Is -1 a valid value?

it comes from `destY`, which is set either by BitBlt inst var access:
`destY = fetchIntOrFloatofObjectifNil(BBDestYIndex, bitBltOop, 0);
`
or thru balloon support:
```
EXPORT(sqInt)
copyBitsFromtoat(sqInt startX, sqInt stopX, sqInt yValue)
{
        destX = startX;
        destY = yValue;

```
We are in the later case:
```
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x321089be150)
  * frame #0: 0x00000001003c58fd Squeak`alphaSourceBlendBits32 at BitBltPlugin.c:820:5
    frame #1: 0x00000001003bf07d Squeak`copyBitsLockedAndClipped at BitBltPlugin.c:1438:3
    frame #2: 0x00000001003b97e6 Squeak`copyBits at BitBltPlugin.c:1257:2
    frame #3: 0x00000001003b9893 Squeak`copyBitsFromtoat(startX=0, stopX=199, yValue=4294967295) at BitBltPlugin.c:1357:2
    frame #4: 0x000000011036eeb4 Squeak3D`b3dDrawSpanBuffer(aet=0x0000000108a313c8, yValue=-1) at b3dMain.c:1146:3
    frame #5: 0x0000000110373d5b Squeak3D`b3dMainLoop(state=0x00000001103aa990, stopReason=0) at b3dMain.c:1448:4
    frame #6: 0x000000011030e1e1 Squeak3D`b3dStartRasterizer at Squeak3D.c:1704:12
    frame #7: 0x00000001000b9cc4 Squeak`primitiveExternalCall at gcc3x-cointerp.c:76948:3

```
up the stack, we have:
```
/* INLINE b3dDrawSpanBuffer(aet, yValue) */
void b3dDrawSpanBuffer(B3DActiveEdgeTable *aet, int yValue)
{
        int leftX, rightX;
        if(aet->size && currentState->spanDrawer) {
                leftX = aet->data[0]->xValue >> B3D_FixedToIntShift;
                rightX = aet->data[aet->size-1]->xValue >> B3D_FixedToIntShift;
                if(leftX < 0) leftX = 0;
                if(rightX > currentState->spanSize) rightX = currentState->spanSize;
                currentState->spanDrawer(leftX, rightX, yValue);
        }
}

```
in b3d.h, we have:
```
        /* Function to call on drawing the output buffer */
        b3dDrawBufferFunction spanDrawer;

```
and what is a `b3dDrawBufferFunction`?
`typedef int (*b3dDrawBufferFunction) (int leftX, int rightX, int yValue);
`
OK, so we get a type mismatch on x64...
We pretend that the function expects a 32bits `int`, when it expects a 64bits `sqInt`...
Bad, because this type mismatch prevents the sign extension...
Parameter is passed on a 64 bit register, and we get the high bits remaining at 0...

So it's a bit more subtle than pointer stored into int.
Frankly, those type mismatch are a plague.
Levente just corrected a bunch of them recently, but when there is such an indirection thru a function pointer, it's getting unobvious...
Especially when we force with a cast:

`../src/plugins//Squeak3D/Squeak3D.c: state.spanDrawer = (b3dDrawBufferFunction) copyBitsFn;
`
```
../src/plugins/Squeak3D/Squeak3D.c:static sqInt copyBitsFn;
../src/plugins/Squeak3D/Squeak3D.c: copyBitsFn = ioLoadFunctionFrom("copyBitsFromtoat", bbPluginName);

```Bah, with all those casts, we let ZERO chance to the compiler to tell us the awfull type mismatch!

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/468#issuecomment-573220454
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/vm-dev/attachments/20200110/5b0dadd8/attachment.html>