[Vm-dev] [OpenSmalltalk/opensmalltalk-vm] Serious character-encoding bug (#449)

[Nicolas Cellier]

Older VM should continue to work with 5.3 image.
The questionnable part is the fallback code that we use in UTF32InputInterpreter whenever (evt->utf32Code) is NULL.
It still perform something like `(evt->charCode) macToSqueak` which is questionable both on Windows and Linux:
- I removed the conversion to macRoman from windows https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/403
- linux still use this macRoman conversion by default, but this can be changed by either vm command line option or environment variable...

I propose that we clean-up after 5.3 release.

------------------------------------------------------
**Some reverse engineering details about unix charCode journey:**

See usage of `sqTextEncoding`global variable:
https://github.com/OpenSmalltalk/opensmalltalk-vm/search?q=sqTextEncoding&unscoped_q=sqTextEncoding

The `charCode` is transformed by `recode` function here:
https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/7875a84c55aaf2ab56ce4b242782b74d7c93d3c7/platforms/unix/vm-display-X11/sqUnixX11.c#L1385

`recode` is used by `x2sqKeyPlain`  `x2sqKeyCompositionInput` and `x2sqKeyInput` functions
https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/7875a84c55aaf2ab56ce4b242782b74d7c93d3c7/platforms/unix/vm-display-X11/sqUnixX11.c#L2078
https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/7875a84c55aaf2ab56ce4b242782b74d7c93d3c7/platforms/unix/vm-display-X11/sqUnixX11.c#L1989
https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/7875a84c55aaf2ab56ce4b242782b74d7c93d3c7/platforms/unix/vm-display-X11/sqUnixX11.c#L1884

Those functions are used as the `x2sqKey` function.
https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/7875a84c55aaf2ab56ce4b242782b74d7c93d3c7/platforms/unix/vm-display-X11/sqUnixX11.c#L261

`x2sqKey`function is used in `handleEvent` function for `KeyPress` and `KeyRelease` events (also mouse wheel, but that's a detail)
https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/7875a84c55aaf2ab56ce4b242782b74d7c93d3c7/platforms/unix/vm-display-X11/sqUnixX11.c#L3757
https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/7875a84c55aaf2ab56ce4b242782b74d7c93d3c7/platforms/unix/vm-display-X11/sqUnixX11.c#L3859

This `keyCode` will finally fill the event structure via `recordKeyboardEvent` function:
https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/aed5e3391301011cc6b9ee6a353ee563f4ab6dbd/platforms/unix/vm/sqUnixEvent.c#L204

Note that `sqTextEncoding` is also used thru this line that define `ux2sqText`
https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/5baff07526c73fdfb813fc8022d88f73faa823c4/platforms/unix/vm/sqUnixCharConv.c#L391


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/449#issuecomment-563957347
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/vm-dev/attachments/20191210/2993206d/attachment.html>