[Vm-dev] SqueakFFI 32-bit (win) | int64_t not working for negative integers

Wed Apr 13 09:56:57 UTC 2022

Hi Nicolas --

Thank you for the advice. Found and fixed the regression
via VMMaker.oscog-mt.3181.

> It's just that we don't have (or don't run ?) enough tests to
> notice the regression.

I found this regression by running the FFI tests in a 32-bit 
build, which is not possible on a recent macOS. I will now
regenerate and push the SqueakFFIPrims sources and
update the "long long" tests in a way that they show this
kind of problem more quickly in the future.

Maybe I should also test 32-bit builds here:
https://github.com/marceltaeumel/squeak-ffi [https://github.com/marceltaeumel/squeak-ffi]

There still seems to be an issue with FFI on macOS.

Best,
Marcel
Am 12.04.2022 22:40:01 schrieb Nicolas Cellier <nicolas.cellier.aka.nice at gmail.com>:
Hi Marcel,
Because there will still be problems passing 64bits (Large) integers to
FFI, such a partial patch has not enough value.
If we pass a 64 bits value, but only use the 32 low bits, the patch rather
tends to obscure things.
I'm pretty sure that passing a LargeInteger to FFI with 64 bit
(signed/unsigned) int type did work back in 2016, so it's not a new
feature, just a regression.
For example we had specific #ffiPushUnsignedLongLongOop:
#ffiPushSignedLongLongOop: at that time. So we would not necessarily go
through the generic coercion #ffiIntegerValueOf:.
Eliot did refactor since, probably so as to support many other
architectures - which is a good thing.
It's just that we don't have (or don't run ?) enough tests to notice the
regression.

The fix would be to resort to some kind of #ffiPushUnsignedLongLongOop:
#ffiPushSignedLongLongOop etc...
All the coercion methods could resort to generic #ffiIntegerValueOf:, but
the 2 above on 32 bits arch.
The C compiler will probably easily factor out the #ffiIntegerValueOf:
common to each case branch on 64 bits arch, we don't have to bother for
handcrafted optimization.
I ain't got much time to push it further, but that's not that hard if
someone want to try it out.

Le mar. 12 avr. 2022 à 15:27, Marcel Taeumel a
écrit :

>
> Hi Nicolas --
>
> Would you mind looking at VMMaker.oscog-mt.3180 in VMMakerInbox?
>
> Do you think this is a good thing to have for the OSVM release? If so, I
> will fix the long-long tests to document this.
>
> Best,
> Marcel
>
> Am 12.04.2022 10:55:33 schrieb Marcel Taeumel :
> Hi Nicolas --
>
> Thanks for the tip. I will have a look.
>
> Do you know whether Squeak FFI 32-bit does (or should) support "long long"
> (or "int64_t") for Large(Positive|Negative)Integer outside the range of
> int32_t?
>
> On 32-bit Squeak, we need both SmallInteger and Large(Positive|Negative)Integer
> to cover int32_t. Not so on 64-bit Squeak. So, I am not sure whether there
> is actually support for int64_t on 32-bit Squeak in this regard...
>
> So, am I looking for a bug or would this rather be a new feature? :-)
>
> Best,
> Marcel
>
> Am 11.04.2022 21:31:58 schrieb Nicolas Cellier <
> nicolas.cellier.aka.nice at gmail.com>:
> Hi Marcel,
> Sounds like 32bits -1 missing a sign extension when converted to 64bits...
> At least, the result is correctly 64bits for sure.
>
> Le lun. 11 avr. 2022 à 18:42, Marcel Taeumel a
> écrit :
>
> >
> > Hi all --
> >
> > This is not working in 32-bit on Windows:
> >
> > FFITestLibrary ffiTest8LongLongSum: -1 with: -1 with: -1 with: -1 with:
> -1
> > with: -1 with: -1 with: -1.
> >
> > It answers " 34359738360 " instead of " - 8 ". It's working fine in a
> > 64-bit build. The debugging output in this test functions says:
> >
> > 8 long longs came in as
> > i1 = 4294967295 (ffffffff)
> > i2 = 4294967295 (ffffffff)
> > i3 = 4294967295 (ffffffff)
> > i4 = 4294967295 (ffffffff)
> > i5 = 4294967295 (ffffffff)
> > i6 = 4294967295 (ffffffff)
> > i7 = 4294967295 (ffffffff)
> > i8 = 4294967295 (ffffffff)
> >
> > ... this is clearly wrong. Or is it? On 64-bit, the same function says:
> >
> > 8 long longs came in as
> > i1 = -1 (ffffffffffffffff)
> > i2 = -1 (ffffffffffffffff)
> > i3 = -1 (ffffffffffffffff)
> > i4 = -1 (ffffffffffffffff)
> > i5 = -1 (ffffffffffffffff)
> > i6 = -1 (ffffffffffffffff)
> > i7 = -1 (ffffffffffffffff)
> > i8 = -1 (ffffffffffffffff)
> >
> > Both are formatted via " %lld (%llx) " I am missing something here...
> did
> > we change the representation of negative integers in Squeak and forgot
> to
> > update something along ThreadedFFIPlugin >> #ffiIntegerValueOf:? :-/
> >
> > Best,
> > Marcel
> >
> > ***
> >
> > EXPORT(long long) ffiTest8LongLongSum(long long c1, long long c2, long
> > long c3, long long c4, long long c5, long long c6, long long c7, long
> long
> > c8)
> > {
> > printf("8 long longs came in as\ni1 = %lld (%llx)\ni2 = %lld (%llx)\ni3
> =
> > %lld (%llx)\ni4 = %lld (%llx)\ni5 = %lld (%llx)\ni6 = %lld (%llx)\ni7 =
> > %lld (%llx)\ni8 = %lld (%llx)\n", c1, c1, c2, c2, c3, c3, c4, c4, c5,
> c5,
> > c6, c6, c7, c7, c8, c8);
> > return c1 + c2 + c3 + c4 + c5 + c6 + c7 + c8;
> > }
> >
> Hi Marcel,
> Sounds like 32bits -1 missing a sign extension when converted to 64bits...
> At least, the result is correctly 64bits for sure.
>
> Le lun. 11 avr. 2022 à 18:42, Marcel Taeumel a
> écrit :
>
>>
>> Hi all --
>>
>> This is not working in 32-bit on Windows:
>>
>> FFITestLibrary ffiTest8LongLongSum: -1 with: -1 with: -1 with: -1 with:
>> -1 with: -1 with: -1 with: -1.
>>
>> It answers " 34359738360 " instead of " - 8 ". It's working fine in a
>> 64-bit build. The debugging output in this test functions says:
>>
>> 8 long longs came in as
>> i1 = 4294967295 (ffffffff)
>> i2 = 4294967295 (ffffffff)
>> i3 = 4294967295 (ffffffff)
>> i4 = 4294967295 (ffffffff)
>> i5 = 4294967295 (ffffffff)
>> i6 = 4294967295 (ffffffff)
>> i7 = 4294967295 (ffffffff)
>> i8 = 4294967295 (ffffffff)
>>
>> ... this is clearly wrong. Or is it? On 64-bit, the same function says:
>>
>> 8 long longs came in as
>> i1 = -1 (ffffffffffffffff)
>> i2 = -1 (ffffffffffffffff)
>> i3 = -1 (ffffffffffffffff)
>> i4 = -1 (ffffffffffffffff)
>> i5 = -1 (ffffffffffffffff)
>> i6 = -1 (ffffffffffffffff)
>> i7 = -1 (ffffffffffffffff)
>> i8 = -1 (ffffffffffffffff)
>>
>> Both are formatted via " %lld (%llx) " I am missing something here... did
>> we change the representation of negative integers in Squeak and forgot to
>> update something along ThreadedFFIPlugin >> #ffiIntegerValueOf:? :-/
>>
>> Best,
>> Marcel
>>
>> ***
>>
>> EXPORT(long long) ffiTest8LongLongSum(long long c1, long long c2, long
>> long c3, long long c4, long long c5, long long c6, long long c7, long long
>> c8)
>> {
>> printf("8 long longs came in as\ni1 = %lld (%llx)\ni2 = %lld (%llx)\ni3 =
>> %lld (%llx)\ni4 = %lld (%llx)\ni5 = %lld (%llx)\ni6 = %lld (%llx)\ni7 =
>> %lld (%llx)\ni8 = %lld (%llx)\n", c1, c1, c2, c2, c3, c3, c4, c4, c5, c5,
>> c6, c6, c7, c7, c8, c8);
>> return c1 + c2 + c3 + c4 + c5 + c6 + c7 + c8;
>> }
>>
>
>

Hi Marcel,
Because there will still be problems passing 64bits (Large) integers to FFI, such a partial patch has not enough value.
If we pass a 64 bits value, but only use the 32 low bits, the patch rather tends to obscure things.

I'm pretty sure that passing a LargeInteger to FFI with 64 bit (signed/unsigned) int type did work back in 2016, so it's not a new feature, just a regression.
For example we had specific #ffiPushUnsignedLongLongOop: #ffiPushSignedLongLongOop: at that time. So we would not necessarily go through the generic coercion #ffiIntegerValueOf:.
Eliot did refactor since, probably so as to support many other architectures - which is a good thing.
It's just that we don't have (or don't run ?) enough tests to notice the regression.

The fix would be to resort to some kind of #ffiPushUnsignedLongLongOop: #ffiPushSignedLongLongOop etc...

All the coercion methods could resort to generic #ffiIntegerValueOf:, but the 2 above on 32 bits arch.
The C compiler will probably easily factor out the #ffiIntegerValueOf: common to each case branch on 64 bits arch, we don't have to bother for handcrafted optimization.

I ain't got much time to push it further, but that's not that hard if someone want to try it out.

Le mar. 12 avr. 2022 à 15:27, Marcel Taeumel <marcel.taeumel at hpi.de [mailto:marcel.taeumel at hpi.de]> a écrit :

Hi Nicolas --

Would you mind looking at VMMaker.oscog-mt.3180 in VMMakerInbox? 

Do you think this is a good thing to have for the OSVM release? If so, I will fix the long-long tests to document this.

Best,
Marcel

Am 12.04.2022 10:55:33 schrieb Marcel Taeumel <marcel.taeumel at hpi.de [mailto:marcel.taeumel at hpi.de]>:

Hi Nicolas --

Thanks for the tip. I will have a look.

Do you know whether Squeak FFI 32-bit does (or should) support "long long" (or "int64_t") for Large(Positive|Negative)Integer outside the range of int32_t?

On 32-bit Squeak, we need both SmallInteger and Large(Positive|Negative)Integer to cover int32_t. Not so on 64-bit Squeak. So, I am not sure whether there is actually support for int64_t on 32-bit Squeak in this regard...

So, am I looking for a bug or would this rather be a new feature? :-)

Best,
Marcel

Am 11.04.2022 21:31:58 schrieb Nicolas Cellier <nicolas.cellier.aka.nice at gmail.com [mailto:nicolas.cellier.aka.nice at gmail.com]>:
Hi Marcel,

Sounds like 32bits -1 missing a sign extension when converted to 64bits...

At least, the result is correctly 64bits for sure.

Le lun. 11 avr. 2022 à 18:42, Marcel Taeumel a

écrit :

>

> Hi all --

>

> This is not working in 32-bit on Windows:

>

> FFITestLibrary ffiTest8LongLongSum: -1 with: -1 with: -1 with: -1 with: -1

> with: -1 with: -1 with: -1.

>

> It answers " 34359738360 " instead of " - 8 ". It's working fine in a

> 64-bit build. The debugging output in this test functions says:

>

> 8 long longs came in as

> i1 = 4294967295 (ffffffff)

> i2 = 4294967295 (ffffffff)

> i3 = 4294967295 (ffffffff)

> i4 = 4294967295 (ffffffff)

> i5 = 4294967295 (ffffffff)

> i6 = 4294967295 (ffffffff)

> i7 = 4294967295 (ffffffff)

> i8 = 4294967295 (ffffffff)

>

> ... this is clearly wrong. Or is it? On 64-bit, the same function says:

>

> 8 long longs came in as

> i1 = -1 (ffffffffffffffff)

> i2 = -1 (ffffffffffffffff)

> i3 = -1 (ffffffffffffffff)

> i4 = -1 (ffffffffffffffff)

> i5 = -1 (ffffffffffffffff)

> i6 = -1 (ffffffffffffffff)

> i7 = -1 (ffffffffffffffff)

> i8 = -1 (ffffffffffffffff)

>

> Both are formatted via " %lld (%llx) " I am missing something here... did

> we change the representation of negative integers in Squeak and forgot to

> update something along ThreadedFFIPlugin >> #ffiIntegerValueOf:? :-/

>

> Best,

> Marcel

>

> ***

>

> EXPORT(long long) ffiTest8LongLongSum(long long c1, long long c2, long

> long c3, long long c4, long long c5, long long c6, long long c7, long long

> c8)

> {

> printf("8 long longs came in as\ni1 = %lld (%llx)\ni2 = %lld (%llx)\ni3 =

> %lld (%llx)\ni4 = %lld (%llx)\ni5 = %lld (%llx)\ni6 = %lld (%llx)\ni7 =

> %lld (%llx)\ni8 = %lld (%llx)\n", c1, c1, c2, c2, c3, c3, c4, c4, c5, c5,

> c6, c6, c7, c7, c8, c8);

> return c1 + c2 + c3 + c4 + c5 + c6 + c7 + c8;

> }

>

Hi Marcel,
Sounds like 32bits -1 missing a sign extension when converted to 64bits...
At least, the result is correctly 64bits for sure.

Le lun. 11 avr. 2022 à 18:42, Marcel Taeumel <marcel.taeumel at hpi.de [mailto:marcel.taeumel at hpi.de]> a écrit :

Hi all --

This is not working in 32-bit on Windows:

FFITestLibrary ffiTest8LongLongSum: -1 with: -1 with: -1 with: -1 with: -1 with: -1 with: -1 with: -1.

It answers " 34359738360 " instead of " - 8 ". It's working fine in a 64-bit build. The debugging output in this test functions says:

8 long longs came in as
i1 = 4294967295 (ffffffff)
i2 = 4294967295 (ffffffff)
i3 = 4294967295 (ffffffff)
i4 = 4294967295 (ffffffff)
i5 = 4294967295 (ffffffff)
i6 = 4294967295 (ffffffff)
i7 = 4294967295 (ffffffff)
i8 = 4294967295 (ffffffff)

... this is clearly wrong. Or is it? On 64-bit, the same function says:

8 long longs came in as
i1 = -1 (ffffffffffffffff)
i2 = -1 (ffffffffffffffff)
i3 = -1 (ffffffffffffffff)
i4 = -1 (ffffffffffffffff)
i5 = -1 (ffffffffffffffff)
i6 = -1 (ffffffffffffffff)
i7 = -1 (ffffffffffffffff)
i8 = -1 (ffffffffffffffff)

Both are formatted via " %lld (%llx) " I am missing something here... did we change the representation of negative integers in Squeak and forgot to update something along ThreadedFFIPlugin >> #ffiIntegerValueOf:? :-/

Best,
Marcel

***

EXPORT(long long) ffiTest8LongLongSum(long long c1, long long c2, long long c3, long long c4, long long c5, long long c6, long long c7, long long c8)
{
printf("8 long longs came in as\ni1 = %lld (%llx)\ni2 = %lld (%llx)\ni3 = %lld (%llx)\ni4 = %lld (%llx)\ni5 = %lld (%llx)\ni6 = %lld (%llx)\ni7 = %lld (%llx)\ni8 = %lld (%llx)\n", c1, c1, c2, c2, c3, c3, c4, c4, c5, c5, c6, c6, c7, c7, c8, c8);
return c1 + c2 + c3 + c4 + c5 + c6 + c7 + c8;
}

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/vm-dev/attachments/20220413/4c3a795d/attachment-0001.html>