Vm-dev June 2007

vm-dev@lists.squeakfoundation.org

20 participants
40 discussions

Clipboard VM crash
by Chris Petsos 04 Jun '07

04 Jun '07

I used this... http://www.squeakvm.org/win32/release/SqueakVM-Win32-3.10.2-bin.zip and the VM crashes when i try to copy the code of a method. To reproduce: 1) Open a Browser. 2) Select the code of any method. 3) Try to copy...boom. Now, the copy/paste method works fine for Workspace data. The crash looks like a faulty memory allocation. Listen what i encountered a few days ago... Many of the methods inside the squeak image have "return character" proper CrLf ( 13 & 10 ). Andreas, i feel that in your VM code you either count less bytes with ClipboardSize or you allocate small buffer for translating to UTF8... I had encountered this when i was implementing the clipboard functionality... take a look at the sqWin32Window.c that i have inside the zip file i've posted. It works OK...It must be a detail you've missed there... Christos.

2 3

Re: New Win32 VM [m17n testers needed]
by Yoshiki Ohshima 04 Jun '07

04 Jun '07

Martin, > >> What I don't understand is: Why do I have to set the language > >> environment (*) to make it work? It's Unicode, so Squeak > >> shouldn't care what the language is. If it needs to know, it > >> should get the language from the system. > > > > Read the Unicode standard. > > I did. What section are you specifically referring to? For example, take a look at this FAQ entry: http://www.unicode.org/faq/han_cjk.html#3 (and one before this and after). > > Because it is Unicode, a mechanism out of scope of Unicode has to > > supply language information to do sensible stuff. > > What is the sensible stuff it needs to do? To display strings in an ok way. http://www.unicode.org/faq/han_cjk.html#2 says that you should select a proper font based on the language you would like to treat the character in. Although the current Squeak implementation is not there yet, you would like to do different sorting or uppercase/lowercase conversions based on the language (even within Latin-1 regions). A segment of text generally should have more information other than the bare code point of Unicode. -- Yoshiki

1 0

Re: New Win32 VM [m17n testers needed]
by Andreas Raab 03 Jun '07

03 Jun '07

Martin v. Löwis wrote: >> Hm ... there isn't any easy way to test this I guess? > > It's actually very easy. > > 1. Create a text file. > 2. rename it to some Cyrillic (or Japanese, or whatever) name > 3. Open it in the listing > > To rename it, the easiest way is to start up charmap.exe, > select a few "funny" characters, copy them to the clipboard, > and past them in explorer into the file name. Ah great. Indeed, there is a series of steps you need to take to make it work: 1) You need to fix LanguageEnvironment class defaultFileNameConverter (make it return UTF8TextConverter new) 2) You need to load a TTF font with the glyphs. For this you need: * Load the TTF loading fixes that Christos posted * Drag and drop a TTF font with the right glyphs on Squeak (Arial works fine) 3) Make this font the default font for text and lists. Once you got all of this the file list shows the correct names. Cheers, - Andreas

1 0

Re: New Win32 VM [m17n testers needed]
by Yoshiki Ohshima 03 Jun '07

03 Jun '07

Martin, > What I couldn't get working is the listing of file names in the > file browser. To reproduce, create file names using Greek, Cyrillic, > and Chinese letters, and then do "open/file list". With the wrong font, > I get question marks. When I select a font that ought to be able to > represent it correctly, I still get a mix of Latin letters and > square boxes. As Andreas wrote, defaultFileNameConverter has to be modified (and the class var in the LanguageEnvironment has to be cleared.) > What I don't understand is: Why do I have to set the language > environment (*) to make it work? It's Unicode, so Squeak > shouldn't care what the language is. If it needs to know, it > should get the language from the system. Read the Unicode standard. Because it is Unicode, a mechanism out of scope of Unicode has to supply language information to do sensible stuff. -- Yoshiki

1 0

Re: New Win32 VM [m17n testers needed]
by Andreas Raab 03 Jun '07

03 Jun '07

Martin v. Löwis wrote: > What I couldn't get working is the listing of file names in the > file browser. To reproduce, create file names using Greek, Cyrillic, > and Chinese letters, and then do "open/file list". With the wrong font, > I get question marks. When I select a font that ought to be able to > represent it correctly, I still get a mix of Latin letters and > square boxes. Digging in the code it seems that the conversion of file names is broken (or at least it seems that way). I can't seem to find the place where a UTF8TextConverter would ever be used (which of course is a requirement for this to work). It seems that the code still assumes that the VMs present file names encoded in the corresponding code pages (which also explains why you'd need to set the language environment etc). The thing to try would be to go into LanguageEnvironment class and change defaultFileNameConverter to include: "Windows VMs always use UTF8-encoded file names now" Smalltalk platformName = 'Win32' ifTrue:[^UTF8TextConverter new]. Cheers, - Andreas

1 0

Re: New Win32 VM [m17n testers needed]
by Andreas Raab 03 Jun '07

03 Jun '07

Martin v. Löwis wrote: > What I couldn't get working is the listing of file names in the > file browser. To reproduce, create file names using Greek, Cyrillic, > and Chinese letters, and then do "open/file list". With the wrong font, > I get question marks. When I select a font that ought to be able to > represent it correctly, I still get a mix of Latin letters and > square boxes. Hm ... there isn't any easy way to test this I guess? The code hasn't changed that much so I would expect this to be working (in particular considering that it seems to work fine for ascii file names). > What I don't understand is: Why do I have to set the language > environment (*) to make it work? It's Unicode, so Squeak > shouldn't care what the language is. If it needs to know, it > should get the language from the system. I don't know. In particular considering that we have now the locale plugin which can detect these settings easily. Cheers, - Andreas

1 0

Re: New Win32 VM [m17n testers needed]
by Andreas Raab 03 Jun '07

03 Jun '07

Yes, I just fixed that. A left-over call to CreateDirectoryA() would make directory creation impossible and later file creation attempts as well. Will be fixed in the next version. Cheers, - Andreas Alexander Lazarevic' wrote: > I just did a quick test out of curiosity. I did an "update code from > server" on a rather old image (Squeak3.9gamma [latest update: #7066]) > and it seems there is an error while creating a cache directory/file. > The primitive "primOpen: fileName writable: writableFlag" fails on the > filename [1] and writableFlag true. > This works with a 3.7.1 VM. > > Alex > > [1] 'C:\Dokumente und > Einstellungen\laza\Desktop\SqueakVM-Win32-3.10.1-bin\package-cache\ScriptLoader-sd.324.mcz' > > Andreas Raab schrieb: >> Hi Folks - >> >> Thanks to some dedicated OLPC-related work done in Greece by Chris >> Petsos[1], we now have a Windows VM with Unicode support enabled. This >> VM will both generate UTF input from characters as well as support >> clipboard, file and directory names in UTF-8. The VM is available here: >> >> http://www.squeakvm.org/win32/release/SqueakVM-Win32-3.10.1-bin.zip >> http://www.squeakvm.org/win32/release/SqueakVM-Win32-3.10.1-src.zip >> >> You are invited to test the new work but be advised that this may >> require some manual adjustments - for an understanding what needs to be >> done, please see [2]. >> >> I'm interested in reports, both good and bad about whether the >> clipboard, file, directory and input support behaves as expected. >> >> [1]http://lists.squeakfoundation.org/pipermail/vm-dev/2007-May/001194.html >> [2]http://lists.squeakfoundation.org/pipermail/vm-dev/2007-June/001306.html >> >> Cheers, >> - Andreas >> > >

1 0

Mac and Unix file/directory/clipboard interface?
by Andreas Raab 03 Jun '07

03 Jun '07

Hi Folks - Since I just went through all of this, can someone explain to me what string encoding the Unix and Mac VMs use for interfacing the file, directory and clipboard functions? If these are all UTF-8 based (which I suspect) then should we just define that *all* strings passed to the VM are to be interpreted as UTF-8 and any VM or function that doesn't deal with UTF-8 correctly is considered broken and needs fixing? It strikes me as a nice, elegant solution to solve this problem once and forever. Comments, anyone? Cheers, - Andreas

3 4

Windows VM Unicode support...
by Chris Petsos 03 Jun '07

03 Jun '07

OK...you can find everyting you need for the Unicode Windows VM here... http://users.kef.sch.gr/chrispetsos/WindowsUnicodeVM.zip I did a few fixes to the code i posted yesterday... The zip has installation notes but I will also paste them here so that anyone can google-find various terms used there... ------------------------------- INSTALLATION NOTES ----------------------------------------- "You should be using the Unicode/UTF8 VM... SqueakUnicodeUTF8.exe" "I tested it on the latest eToys image for OLPC." "Let's start..." "Drag 'n Drop UnicodeEnvironment.cs -> fileIn entire file" "Drag 'n Drop GreekEnvironment.cs -> fileIn entire file." "Execute this*" Locale currentPlatform: (Locale localeID: (LocaleID isoString: 'el')). "Drag 'n Drop TTFUnicodeFix.cs -> fileIn entire file." "Drag 'n Drop your favourite TT Unicode font." "Now you should have a new font installed." "Highlight a text." "Hit Alt+k and select your new font" " Well...that was it...type in your locale's specific unicode characters and see them getting rendered with a beautiful TT unicode font. " "Now, this version should be working OK for other languages too." "Please feel free to test it for other languages and other image versions too..." " So, now... 1) you have a VM that is sending unicode chars to eToys (thanks to Diomidis Spinellis...) 2) you have a fixed Unicode font support (thanks to Bert Freudenberg...) 3) you have UTF-8 clipboard functionality (thanks to me!...and all the Vm-dev support @:) ) 4) you have a unicode environment inside the image (thanks to me again of course!) ;-) " " Ok...enough with happy talking...any problems that you'll encounter (hope you won't) please post them up... " "*To execute: (Highlight -> Alt+d)" ------------------------------- INSTALLATION NOTES ----------------------------------------- Christos.

3 5

Unicode clipboard in Windows VM...
by Chris Petsos 01 Jun '07

01 Jun '07

Ok guys... this is my solution for unicode clipboard support in Windows VM... It works fine... I can make some improvements in memory management but it's ok for now... /****************************************************************************/ /* Clipboard */ /****************************************************************************/ int clipboardSize(void) { HANDLE h, h2; WCHAR *src; unsigned char *csrc; int len, bytesNeeded; // Do we have text in the clipboard? if(!IsClipboardFormatAvailable(CF_TEXT)) return 0; if(!OpenClipboard(stWindow)) return 0; // Get it in unicode format. h = GetClipboardData(CF_UNICODETEXT); src = GlobalLock(h); // How many bytes do we want to store those unicode chars in UTF8 format? bytesNeeded = WideCharToMultiByte( CP_UTF8, 0, src, -1, NULL, 0, NULL, NULL ); h2 = GlobalAlloc(GMEM_MOVEABLE | GMEM_DDESHARE, bytesNeeded); csrc = GlobalLock(h2); // Convert Unicode text to UTF8. WideCharToMultiByte( CP_UTF8, 0, src, -1, csrc, bytesNeeded, NULL, NULL ); // Consider CrLf as a 1-byte character. len = bytesNeeded; while(len--) if(((*csrc == 13) && (csrc[1] == 10))) { bytesNeeded--; csrc++; } else { csrc++; } // Clipboard size is bytesNeeded excluding the terminating character. len = bytesNeeded - 1; GlobalUnlock(h); GlobalUnlock(h2); return len; } /* send the given string to the clipboard */ int clipboardWriteFromAt(int count, int byteArrayIndex, int startIndex) { HANDLE h, h2; unsigned char *src; int wcharsNeeded, len; WCHAR *out, *tmp; if(!OpenClipboard(stWindow)) return 0; // Get the pointer to the byte array. src = (unsigned char *)byteArrayIndex + startIndex; // How many WCHARs do we need to store the UTF8 represented bytes from Squeak? wcharsNeeded = MultiByteToWideChar( CP_UTF8, 0, src, count+1, NULL, 0 ); // If we have Cr only "returns" then we will need another character for Lf. len = wcharsNeeded * sizeof(WCHAR); while(len--) if ((*src++ == 13) && (*src != 10)) wcharsNeeded++; // Point to start of byte aray. src = (unsigned char *)byteArrayIndex + startIndex; // Allocate needed memory for wcharsNeeded WCHARs. h = GlobalAlloc(GMEM_MOVEABLE | GMEM_DDESHARE, wcharsNeeded * sizeof(WCHAR)); out = GlobalLock(h); // Convert them to Unicode UTF16. MultiByteToWideChar( CP_UTF8, 0, src, count+1, out, wcharsNeeded ); // Allocate needed memory for intermediate buffer. // OK, i could do some more work here in order to avoid a whole new buffer just for // fixing the CrLf thing, but it's ok for now... h2 = GlobalAlloc(GMEM_MOVEABLE | GMEM_DDESHARE, wcharsNeeded * sizeof(WCHAR)); tmp = GlobalLock(h2); // Add missing Lf characters... len = wcharsNeeded; while(len--) if((*out == 13)) { /* special case: crlf translation */ *tmp = 13; tmp++; *tmp = 10; tmp++; out++; } else { /* regular case: lookup translation */ *tmp = *out; tmp++; out++; } EmptyClipboard(); GlobalUnlock(h); GlobalUnlock(h2); // Send the Unicode text to the clipboard. SetClipboardData(CF_UNICODETEXT, h2); *src=0; *tmp=0; /* Note: After setting clipboard data, the memory block previously allocated belongs to the clipboard - not to the app. */ CloseClipboard(); return 1; } /* transfer the clipboard data into the given byte array */ int clipboardReadIntoAt(int count, int byteArrayIndex, int startIndex) { HANDLE h, h2; unsigned char *dst, *tmp; WCHAR *src; int tel,len, bytesNeeded; if(!IsClipboardFormatAvailable(CF_TEXT)) /* check for format CF_TEXT */ return 0; if(!OpenClipboard(stWindow)) return 0; // Get clipboard data in Unicode format h = GetClipboardData(CF_UNICODETEXT); src = GlobalLock(h); // How many bytes do we want to store the Unicode chars to UTF8 representation? bytesNeeded = WideCharToMultiByte( CP_UTF8, 0, src, -1, tmp, 0, NULL, NULL ) + 1; h2 = GlobalAlloc(GMEM_MOVEABLE | GMEM_DDESHARE, bytesNeeded); tmp = GlobalLock(h2); // Convert Unicode text to UTF8. WideCharToMultiByte( CP_UTF8, 0, src, -1, tmp, bytesNeeded, NULL, NULL); len = bytesNeeded; // Get the pointer to the byte array. dst= (unsigned char *)byteArrayIndex + startIndex; while(len--) if(((*tmp == 13) && (tmp[1] == 10))) { /* RvL 17-04-1998 drop the line feed after a carriage return but leave lone line feeds alone may transfer less than 'len' bytes but who cares i.e. if they use the clipboardSize there should be no problem.*/ *dst = 13; dst++; tmp++; tmp++; } else { /* regular case: lookup translation */ *dst = *tmp; dst++; tmp++; } GlobalUnlock(h); GlobalUnlock(h2); CloseClipboard(); return bytesNeeded; } The only thing is that in the image side you need a UTF8TextConverter before sending text to the clipboard or getting from it...so, my unicode clipboard interpreter has these methods... fromSystemClipboard: aString ^ aString convertFromWithConverter: (UTF8TextConverter new). toSystemClipboard: aString | result utfString | " aString isOctetString ifTrue: [^ aString asOctetString]." utfString := aString convertToWithConverter: (UTF8TextConverter new). result _ WriteStream on: (String new: utfString size). utfString do: [:each | result nextPut: each]. ^ result contents. I am pretty sure it will need some minor fixes... I'll post links to ready made builds of this version so others can try it too, soon. Any comments, suggestions, C-related fixes are most welcome... Christos

2 2

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Vm-dev June 2007