Oh, interesting. That reminds me of the fix that I needed for Windows, let me see... yes, here it is: LanguageEnvironment class>>defaultFileNameConverter needs to be fixed since it is (wrongly) guessing the file name encoding based on the currently active locale (which makes no sense btw, since the locale doesn't mean Jack for file name encodings). Hm ... lemme try this ... ah, interesting. It appears that I can make the Umlauts work on Unix correctly if and only if: * I fix the above method to return UTF8TextConverter in every case [*1] * I use -pathenc MacRoman -textenc MacRoman Which makes no sense to me since neither the path nor the text encoding is MacRoman but it appears to work. Huh?

[*1] And that of course reminds me that nobody has really made any comment on why the hell we still deal with all of these nonsensical legacy encodings and don't just go straight to UTF-8 in the VM interface which would simplify *lots* of cruft in the code.

Cheers, - Andreas

John M McIntosh wrote:

On Dec 29, 2007, at 11:32 PM, Andreas Raab wrote:

...

Hi -

Due to a bug reported against Qwaq Forums I needed to look into how the Unix VM encodes file and path names and got terribly confused.

Also see

From:     bert@impara.de
Subject:     Re: mac carbon VM goes to unix file names, testers needed

Date:     February 6, 2006 12:54:17 AM PST (CA)

...

We've been using UTF-8 VM encoding for a while, to be able to access files with non-ASCII characters in their path. However, I haven't found a way to permanently switch the image (3.8) to UTF-8 encoding. It keeps resetting to Latin1 on startup. The only solution for me was to put this line in my own startup code:

LanguageEnvironment classPool at: #FileNameConverterClass put: 

UTF8TextConverter

Did anybody create a better solution than this horrible hack? I feel someone else must be using UTF-8, too, now that we support it ...

• Bert -

--

John M. McIntosh johnmci@smalltalkconsulting.com Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ===========================================================================

On Dec 30, 2007, at 1:28 AM, Andreas Raab wrote:

Oh, interesting. That reminds me of the fix that I needed for Windows, let me see... yes, here it is: LanguageEnvironment class>>defaultFileNameConverter needs to be fixed since it is (wrongly) guessing the file name encoding based on the currently active locale (which makes no sense btw, since the locale doesn't mean Jack for file name encodings). Hm ... lemme try this ... ah, interesting. It appears that I can make the Umlauts work on Unix correctly if and only if:

• I fix the above method to return UTF8TextConverter in every case

[*1]

• I use -pathenc MacRoman -textenc MacRoman

Which makes no sense to me since neither the path nor the text encoding is MacRoman but it appears to work. Huh?

Careful now is that

ü 0x9F in macroman or 0x000000FC in utf-32

which is coming back from the VM, then what does the LanguageEnvironment class>>defaultFileNameConverter do and which FONT and character set does that font think it lives in? Since the hex value might not have proper representation if you use a non-unicode proper font after converting from something to unicode....

-- = = = ======================================================================== John M. McIntosh johnmci@smalltalkconsulting.com Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com = = = ========================================================================

I think all the vertical squeak packages (scratch, sophie, plopp) use UTF8 and get to ignore the file tools. And don't get Tim started on what he thinks of the classes that deal with files...

On irc yesterday I learned that Rio has at least one fan!

Keith

On Dec 30, 2007, at 2:11 AM, Yoshiki Ohshima wrote:

...

Hm ... lemme try this ... ah, interesting. It appears that I can make the Umlauts work on Unix correctly if and only if:

• I fix the above method to return UTF8TextConverter in every case

[*1]

• I use -pathenc MacRoman -textenc MacRoman

Which makes no sense to me since neither the path nor the text encoding is MacRoman but it appears to work. Huh?

Yes, on Unix VM, another historical mishappen caused it; "MacRoman" still means "no conversion" so that if the image passes UTF-8 string, the UTF-8 string is passed to system calls.

Er, well I'm not sure that's quite accurate? In looking at sqUnixCharConv.c it seems to say that if the text encoding is macroman and the path encoding is macroman the translation from unix path to squeak would be macroman to macroman so nothing would happen.

Convert(sq,ux, Path, sqTextEncoding, uxPathEncoding, 1, 0); // normalised paths for HFS+ Convert(ux,sq, Path, uxPathEncoding, sqTextEncoding, 0, 0);

in sqInt dir_Lookup(char *pathString, sqInt pathStringLength, sqInt index, /* outputs: */ char *name, sqInt *nameLength, sqInt *creationDate, sqInt *modificationDate, sqInt *isDirectory, squeakFileOffsetType *sizeIfFile)

we find *nameLength= ux2sqPath(dirEntry->d_name, nameLen, name, MAXPATHLEN, 0);

However I note the carbon vm does if (norm) // HFS+ imposes Unicode2.1 decomposed UTF-8 encoding on all path elements CFStringNormalize(str, kCFStringNormalizationFormD); // canonical decomposition else CFStringNormalize(str, kCFStringNormalizationFormC); // pre- combined

but the unix VM does not do this, which I think is an error based on:

See From: tetha@st.rim.or.jp Subject: [Vm-dev] Patch for filename normalization of mac vm

Date: March 11, 2007 8:21:06 PM PDT (CA)

To: vm-dev@lists.squeakfoundation.org

Hi,

I've found the latest mac vm (or recent version) fails to normalize UTF file name. It seems to be the function convertChars() of sqMacUnixFileInterface.c, which normalizes only decompose when converting squeak string to unix, but I think it needs pre-combined when unix string to squeak, and I noticed normalization form should be canonical (exactly should be kCFStringNormalizationFormC) for pre- combined.

Patch (diff format of xcode tool) for this problem is attached to this mail.

Regards,

Tetsuya HAYASHI, tetha@st.rim.or.jp, tetha@mac.com

PS I note if you feed CFStringCreateWithBytes bad data, why it returns NULL, then the lurking CFStringCreateMutableCopy core dumps the VM. That's why I check for it in the carbon vm. Normally you won't see this issue unless you get creative... -- = = = ======================================================================== John M. McIntosh johnmci@smalltalkconsulting.com Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com = = = ========================================================================

On Dec 30, 2007, at 4:09 AM, Andreas Raab wrote:

Yoshiki Ohshima wrote:

...

...

Hm ... lemme try this ... ah, interesting. It appears that I can make the Umlauts work on Unix correctly if and only if:

• I fix the above method to return UTF8TextConverter in every case

[*1]

• I use -pathenc MacRoman -textenc MacRoman

Which makes no sense to me since neither the path nor the text encoding is MacRoman but it appears to work. Huh?

Yes, on Unix VM, another historical mishappen caused it; "MacRoman" still means "no conversion" so that if the image passes UTF-8 string, the UTF-8 string is passed to system calls.

Playing around a little it appears as if the Unix VM always converts path names with the assumption that Squeak uses MacRoman in the image and only -pathenc affects the translation between file system and the image (i.e., -textenc has *no* effect on path name translation whatsoever). Can someone confirm this? It would explain why -pathenc MacRoman works (since like you say it's really the "no conversion" flag) if combined with a proper file name converter in the image.

Mmm for the -pathenc and the -textenc from what I can see the data coming from the file system is said to exist in the form -pathenc and translated to a CFString in UTF-32, then translated back to a byte string in -textenc.

In sending the data to the file system, it said it exists in the form - textenc, then translated to a CFString in UTF-32 then CFStringNormalize(str, kCFStringNormalizationFormD); // canonical decomposition, then translated back to a byte string in -pathenc.

I'll note the kCFStringNormalizationFormD operation (and all above/ below) only occurs if this is macintosh. If this is a Linux/BSD unix system then iconv is used. So is this on a mac or some Linux/BSD system?

That and I think a kCFStringNormalizationFormC is needed in the first step to properly compose the characters.

For background

Ok, let's see in the mac carbon vm we get back from the file system for LATIN CAPITAL LETTER SCHWA + LATIN CAPITAL LETTER A WITH ACUTE ƏÁ.png

0xC6, 0x8F, A, 0xCC, 0x81, .png Note how the A0xCC81 is the decomposed UTF8, this is what is stored in the HFS+ file system.

We convert that from UTF8 to the target of MacRoman for path names by default in the base carbon VM. This means converting to a CFString from kCFStringEncodingUTF8 then applying CFStringNormalize(str, kCFStringNormalizationFormC); // pre-combined then pulling back the bytes as MacRoman, that becomes

?, 0xE7, .png

since the translation of the Ə from utf8 to macroman fails, but the E7 is correct macroman for the Á

Now if I set the vm up to use UTF8 as the path name default.

after we apply the kCFStringNormalizationFormC step and pull back the data as UTF8 it is

0xC6, 0x8F, 0xC3, 0x81, .png where the 0xC381 is the (LATIN CAPITAL LETTER A WITH ACUTE) in UTF8 or 0x00c1 in utf-16 or 0x000000c1 in utf-32

Now if I remove the CFStringNormalize(str, kCFStringNormalizationFormC); // pre-combined which does not exist in the base unix vm, then I get back

0xC6, 0x8F, A, 0xCC, 0x81, '.png'

which is the decomposed UTF8. I'll note in the file browser it shows as Æ∑AÌ∞.png

but it does work....

NOW the question is what does the translation do... mmm Well if I try LanguageEnvironment classPool at: #FileNameConverterClass put: UTF8TextConverter then it shows: ?A?.png

which is mmm, less wrong? But it does work.

Now if in the base unix VM you take path encoding and text encoding and set to macroman Convert(ux,sq, Path, uxPathEncoding, sqTextEncoding, 0, 0); then we are saying the operating system (unix) path encoding is macroman and the squeak path name encoding is macroman that gives back

0xC6, 0x8F, A, 0xCC, 0x81, '.png'

since as thought the macroman to macroman translation does nothing.

However the UTF8TextConverter does not work with decomposed UTF8 so what the user would see is not correct, since it assumes we are working with precomposed UTF8 when it converts it to UTF-32 for the font system's enjoyment.

I suspect to fix properly a CFStringNormalize(str, kCFStringNormalizationFormC); is needed in the Unix sqUnixCharConv.c and applied when applicable.

Notes from Apple's site

For example, an Á (A acute) can be encoded either precomposed, as U +00C1 (LATIN CAPITAL LETTER A WITH ACUTE), or decomposed, as U+0041 U +0301 (UTF8 is 0xCC81) (LATIN CAPITAL LETTER A followed by a COMBINING ACUTE ACCENT). Precomposed characters are more common in the Windows world, whereas decomposed characters are more common on the Mac.

-- = = = ======================================================================== John M. McIntosh johnmci@smalltalkconsulting.com Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com = = = ========================================================================

Yoshiki Ohshima wrote:

...

What encoding options are being used on OLPC? Do non-ascii file names, clipboard, drag and drop etc. work on OLPC?

Filenames are in UTF-8. Clipboard and Drag and drop uses somewhat strange mechanism, but they are normalized to UTF-8 as well.

Odd. Checking the latest OLPC image I have access to (etoys2.3-1867) there is nothing I can see that would deal properly in the image with non-ascii file names on Unix. LanguageEnvironment still returns "self currentPlatform class fileNameConverterClass" which won't use an UTF-8 converter anywhere. So unless I'm missing something it probably shouldn't work on OLPC. Maybe try it with some "real" UTF-32 names?

Cheers, - Andreas