[Vm-dev] [OpenSmalltalk/opensmalltalk-vm] BitBlt Segmentation Fault with Magic (2^31) value (#447)

[Nicolas Cellier]

The signed integer overflow reported by -fsanitize is not related, what is related is the assert warning: 

```
(((usqInt)destIndex)) < endOfDestination 2226
Process 77080 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0xffffffff0d50d768)
    frame #0: 0x000000010048bc51 Squeak`copyLoopNoSource at BitBltPlugin.c:2227:14
   2224			}
   2225			destMask = mask1;
   2226			assert((((usqInt)destIndex)) < endOfDestination);
-> 2227			destWord = long32At(destIndex);
   2228			mergeWord = mergeFnwith(halftoneWord, destWord);
   2229			destWord = (destMask & mergeWord) | (destWord & ((unsigned int)~destMask));
   2230			long32Atput(destIndex, destWord);
Target 0: (Squeak) stopped.
```

```
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0xffffffff0d50d768)
  * frame #0: 0x000000010048bc51 Squeak`copyLoopNoSource at BitBltPlugin.c:2227:14
    frame #1: 0x0000000100481b71 Squeak`copyBitsLockedAndClipped at BitBltPlugin.c:1505:3
    frame #2: 0x000000010047b6e6 Squeak`copyBits at BitBltPlugin.c:1257:2
    frame #3: 0x000000010047ba23 Squeak`primitiveCopyBits at BitBltPlugin.c:5128:2
    frame #4: 0x0000000108a01670
    frame #5: 0x0000000100001a65 Squeak`interpret at gcc3x-cointerp.c:2772:3
    frame #6: 0x00000001003abd87 Squeak`-[sqSqueakMainApplication runSqueak](self=0x0000000101867010, _cmd="runSqueak") at sqSqueakMainApplication.m:201:2

(lldb) p/x destIndex
(usqInt) $8 = 0xffffffff0d50d768
(lldb) p/x endOfDestination
(usqInt) $9 = 0x000000010d741768
(lldb) p/x destX
(sqInt) $32 = 0x0000000080000000
(lldb) p/x dx
(int) $36 = 0x80000000
(lldb) print dx
(int) $34 = -2147483648
```

`destX` is 2^31, since it is on 64bits (signed), there is no overflow at this stage.<br>
The overflow is on `dx` which is an `int`.<br>
It seems that `-fsanitize=undefined` does not signal case of overflow unsigned->signed, because it is not UB...

One solution is to change the type of dx and dy to sqInt.
Then, with 64 bits, it might still be possible to craft a value that will overflow, this will have to be reviewed...

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/447#issuecomment-557294695
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/vm-dev/attachments/20191121/f44d0352/attachment.html>