[Vm-dev] [OpenSmalltalk/opensmalltalk-vm] 015d38: Fix some Squeak3D UB: shifting left some negative int

[Nicolas Cellier]

  Branch: refs/heads/Cog
  Home:   https://github.com/OpenSmalltalk/opensmalltalk-vm
  Commit: 015d381da7b553f0add8aa53b3f72014b16f5c82
      https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/015d381da7b553f0add8aa53b3f72014b16f5c82
  Author: Nicolas Cellier <nicolas.cellier.aka.nice at gmail.com>
  Date:   2020-02-08 (Sat, 08 Feb 2020)

  Changed paths:
    M platforms/Cross/plugins/Squeak3D/b3dDraw.c
    M platforms/Cross/plugins/Squeak3D/b3dMain.c

  Log Message:
  -----------
  Fix some Squeak3D UB: shifting left some negative int

A reproducible case of crash provided by Stephane Rollandin gives the following warning with clang `-fsanitize=undefined`:

>../../platforms/Cross/plugins/Squeak3D/b3dMain.c:1252:29: runtime error: left shift of negative value -760
>../../platforms/Cross/plugins/Squeak3D/b3dMain.c:1254:25: runtime error: left shift of negative value -751
>../../platforms/Cross/plugins/Squeak3D/b3dDraw.c:317:33: runtime error: left shift of negative value -802
>../../platforms/Cross/plugins/Squeak3D/b3dDraw.c:318:33: runtime error: left shift of negative value -802
>../../platforms/Cross/plugins/Squeak3D/b3dDraw.c:316:33: runtime error: left shift of negative value -114
>../../platforms/Cross/plugins/Squeak3D/b3dMain.c:829:61: runtime error: left shift of negative value -2

On OSX optimized VM, a crash happens in b3dMain.c, in function b3dAddBackFill at line 994 soon after those warnings
By protecting the shift with (unsigned) cast, this particular crash disappear.

There is still other crash happening related to bad fill list, but one thing at a time...