[Vm-dev] VM crash in the Squeak3D plugin, the sequeal

Nicolas Cellier nicolas.cellier.aka.nice at gmail.com
Fri Jan 10 21:06:45 UTC 2020


Hi all,
and I probably broke it with
https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/448
The function signature was already BADLY unconsistent!
But luckily, the erroneous 64 bits arguments did get copied into 32 bits
dstY before my fix...
I sort of broke the spell and the magic vanished ;)
Brittle code = hazardous life

What I'm very unsure of now, is if writing at negative offset is a good
idea (buffer underflow?), or if this uncovers yet another bug at upper
level...

Le jeu. 9 janv. 2020 à 23:12, Nicolas Cellier <
nicolas.cellier.aka.nice at gmail.com> a écrit :

> I note a few bad things:
>
> * thread #1, queue = 'com.apple.main-thread', stop reason = Invalid shift
> base
>   * frame #0: 0x0000000100ad5f40
> libclang_rt.ubsan_osx_dynamic.dylib`__ubsan_on_report
>     frame #1: 0x0000000100ad05f8
> libclang_rt.ubsan_osx_dynamic.dylib`__ubsan::Diag::~Diag() + 216
>     frame #2: 0x0000000100ad32d1
> libclang_rt.ubsan_osx_dynamic.dylib`handleShiftOutOfBoundsImpl(__ubsan::ShiftOutOfBoundsData*,
> unsigned long, unsigned long, __ubsan::ReportOptions) + 929
>     frame #3: 0x0000000100ad2f24
> libclang_rt.ubsan_osx_dynamic.dylib`__ubsan_handle_shift_out_of_bounds + 68
>     frame #4: 0x000000011036fb5c
> Squeak3D`b3dMainLoop(state=0x00000001103aa990, stopReason=0) at
> b3dMain.c:1249:29
> ../../platforms/Cross/plugins/Squeak3D/b3dMain.c:1249:29: runtime error:
> left shift of negative value -8
>
> same at following lines
>     frame #4: 0x000000011036fbca
> Squeak3D`b3dMainLoop(state=0x00000001103aa990, stopReason=0) at
> b3dMain.c:1251:25
>     frame #5: 0x0000000110373c6a
> Squeak3D`b3dMainLoop(state=0x00000001103aa990, stopReason=0) at
> b3dMain.c:1428:8
>     frame #5: 0x0000000110373c6a
> Squeak3D`b3dMainLoop(state=0x00000001103aa990, stopReason=0) at
> b3dMain.c:1428:8
> Don't you rely on UB !
>
> Then at crash site:
> Process 95068 stopped
> * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS
> (code=1, address=0x321089be150)
>     frame #0: 0x00000001003c58fd Squeak`alphaSourceBlendBits32 at
> BitBltPlugin.c:820:5
>    817 sourceWord = long32At(srcIndex);
>    818 srcAlpha = ((usqInt) sourceWord) >> 24;
>    819 if (srcAlpha == 0xFF) {
> -> 820 long32Atput(dstIndex, sourceWord);
>    821 srcIndex += 4;
>
> we have :
> (lldb) p/x dstIndex
> (sqInt) $9 = 0x00000321089be150
> (lldb) p/x destBits
> (sqInt) $14 = 0x00000001089be470
>
> Wow, dstIndex is very far from destBits... because:
> (lldb) p/x dy
> (sqInt) $13 = 0x00000000ffffffff
> (lldb) p/x dstY
> (sqInt) $15 = 0x00000000ffffffff
>
> Hmm is that intended to be so large? or just -1?
> Is it a missing sign extension?
> Is -1 a valid value?
>
> it comes from destY, which is set either by BitBlt inst var access:
> destY = fetchIntOrFloatofObjectifNil(BBDestYIndex, bitBltOop, 0);
>
> or thru balloon support:
> EXPORT(sqInt)
> copyBitsFromtoat(sqInt startX, sqInt stopX, sqInt yValue)
> {
>         destX = startX;
>         destY = yValue;
>
> We are in the later case:
> (lldb) bt
> * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS
> (code=1, address=0x321089be150)
>   * frame #0: 0x00000001003c58fd Squeak`alphaSourceBlendBits32 at
> BitBltPlugin.c:820:5
>     frame #1: 0x00000001003bf07d Squeak`copyBitsLockedAndClipped at
> BitBltPlugin.c:1438:3
>     frame #2: 0x00000001003b97e6 Squeak`copyBits at BitBltPlugin.c:1257:2
>     frame #3: 0x00000001003b9893 Squeak`copyBitsFromtoat(startX=0,
> stopX=199, yValue=4294967295) at BitBltPlugin.c:1357:2
>     frame #4: 0x000000011036eeb4
> Squeak3D`b3dDrawSpanBuffer(aet=0x0000000108a313c8, yValue=-1) at
> b3dMain.c:1146:3
>     frame #5: 0x0000000110373d5b
> Squeak3D`b3dMainLoop(state=0x00000001103aa990, stopReason=0) at
> b3dMain.c:1448:4
>     frame #6: 0x000000011030e1e1 Squeak3D`b3dStartRasterizer at
> Squeak3D.c:1704:12
>     frame #7: 0x00000001000b9cc4 Squeak`primitiveExternalCall at
> gcc3x-cointerp.c:76948:3
>
> up the stack, we have:
> /* INLINE b3dDrawSpanBuffer(aet, yValue) */
> void b3dDrawSpanBuffer(B3DActiveEdgeTable *aet, int yValue)
> {
>         int leftX, rightX;
>         if(aet->size && currentState->spanDrawer) {
>                 leftX = aet->data[0]->xValue >> B3D_FixedToIntShift;
>                 rightX = aet->data[aet->size-1]->xValue >>
> B3D_FixedToIntShift;
>                 if(leftX < 0) leftX = 0;
>                 if(rightX > currentState->spanSize) rightX =
> currentState->spanSize;
>                 currentState->spanDrawer(leftX, rightX, yValue);
>         }
> }
>
> in b3d.h, we have:
>         /* Function to call on drawing the output buffer */
>         b3dDrawBufferFunction spanDrawer;
>
> and what is a b3dDrawBufferFunction?
> typedef int (*b3dDrawBufferFunction) (int leftX, int rightX, int yValue);
>
> OK, so we get a type mismatch on x64...
> We pretend that the function expects a 32bits int, when it expects a
> 64bits sqInt...
> Bad, because this type mismatch prevents the sign extension...
> Parameter is pased on a 64 bit register, and we get the high bits
> remaining at 0...
>
> So it's a bit more subtle than pointer stored into int.
> Frankly, those type mismatch are a plague.
> Levente just corrected a bunch of them recently, but when there is such an
> indirection thru a function pointer, it's getting unobvious...
> Especially when we force with a cast:
>
> ../src/plugins//Squeak3D/Squeak3D.c: state.spanDrawer =
> (b3dDrawBufferFunction) copyBitsFn;
>
> ../src/plugins/Squeak3D/Squeak3D.c:static sqInt copyBitsFn;
> ../src/plugins/Squeak3D/Squeak3D.c: copyBitsFn =
> ioLoadFunctionFrom("copyBitsFromtoat", bbPluginName);
>
> Bah, with all those casts, we let ZERO chance to the compiler to tell us
> the awfull type mismatch!
>
> Le jeu. 9 janv. 2020 à 22:18, Phil B <pbpublist at gmail.com> a écrit :
>
>>
>> I can confirm that at least basic functionality of the plugin is working
>> on 64-bit x86 Linux (and 32-bit x86 and ARM) since I use it almost daily.
>> However I'm only using it to create/manage the 3D viewport... everything
>> else I'm doing via FFI. (i.e. if there is other functionality that is
>> broken, I likely wouldn't be seeing it)
>>
>> On Thu, Jan 9, 2020 at 11:48 AM Eliot Miranda <eliot.miranda at gmail.com>
>> wrote:
>>
>>>
>>> Hi Dave,
>>>
>>> On Thu, Jan 9, 2020 at 6:04 AM David T. Lewis <lewis at mail.msen.com>
>>> wrote:
>>>
>>>>
>>>> On Thu, Jan 09, 2020 at 12:57:23PM +0100, St??phane Rollandin wrote:
>>>> >
>>>> > Hello all,
>>>> >
>>>> > I have had a report that the code for my Balloon3D-based game (at
>>>> > http://www.zogotounga.net/comp/guardians.htm) does not work at all
>>>> on
>>>> > 64-bit images.
>>>> >
>>>> > The report was about a linux system, and indeed I could confirm that
>>>> it
>>>> > also applies on Windows.
>>>> >
>>>> > I have uploaded a ready-to-crash trunk image here, with instructions:
>>>> > http://www.zogotounga.net/swap/crash-64.zip
>>>> >
>>>> > Note that no crash dump is produced.
>>>>
>>>>
>>>> Hi Stef,
>>>>
>>>> I am fairly sure that the Squeak3D plugin was never updated for 64-bit
>>>> platforms, so it probably will not work on any 64-bit VM.
>>>>
>>>> Updating a plugin like this mostly requires finding and fixing all the
>>>> places where pointer values are assigned to C int variables. For most
>>>> of the plugins, this involved some changes to both the platform support
>>>> code (C code in git) as well as to the plugin in VMMaker.
>>>>
>>>> Is anyone interested in taking on a worthwhile VM project with
>>>> manageable
>>>> scope? Updating the Squeak3D plugin would be a good project. I can give
>>>> some tips and advice if anyone wants to give it a try.
>>>>
>>>
>>> The Squeak3D polygon was updated for 64 bits. It is built and included
>>> as an external plugin on all 32 bit and 64 bit Spur VMs. It builds with
>>> only one (expected) warning:
>>>
>>> ../../src/plugins/Squeak3D/Squeak3D.c:6:13: warning: unused variable
>>> '__buildInfo' [-Wunused-variable]
>>> 1 warning generated.
>>>
>>> So whatever the issue is is deeper than simply not having been updated.
>>>
>>> _,,,^..^,,,_
>>> best, Eliot
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/vm-dev/attachments/20200110/1d935867/attachment-0001.html>


More information about the Vm-dev mailing list