Am 26.03.2006 um 09:11 schrieb Andreas Raab:

tim Rowledge wrote:

...

On 25-Mar-06, at 10:33 PM, Andreas Raab wrote:

...

This function is news to me - can somebody explain what it's good for, where it should be used and (most importantly) why we'd export this for plugins but don't use in the VM itself (e.g., no senders)?

See mantis 2044 for some details and the original rationale. It also relates somewhat to :-

I guess you mean http://bugs.impara.de/view.php?id=2404 yes? One question about this: Do we have a way of doing that kind of file name conversion from the image itself? It strikes me as somewhat odd that a plugin needs to do that conversion given that the image can just pass a well-formed version of the string as the argument, no?

Not if the VM creates the illusion of some other naming than the actual system uses. Like the use of HFS file names in the Mac VM (with : separators), where the VM translates to unix path names nowadays. The problem with the macro was that it called (on the Mac at least) a non-exported VM function. But in the freetype plugin I needed to be able to convert from the path name the image uses to the actual system path name. So just using the macro didn't work, which is why we added that function.

- Bert -

On 25-Mar-06, at 11:11 PM, Andreas Raab wrote:

tim Rowledge wrote:

...

On 25-Mar-06, at 10:33 PM, Andreas Raab wrote:

...

This function is news to me - can somebody explain what it's good for, where it should be used and (most importantly) why we'd export this for plugins but don't use in the VM itself (e.g., no senders)?

See mantis 2044 for some details and the original rationale. It also relates somewhat to :-

I guess you mean http://bugs.impara.de/view.php?id=2404 yes?

Yup. Meant to fix that before hitting send but screwed up, obviously.

One question about this: Do we have a way of doing that kind of file name conversion from the image itself? It strikes me as somewhat odd that a plugin needs to do that conversion given that the image can just pass a well-formed version of the string as the argument, no?

Actually, no. Until and unless we ever get some decent file handling code in the image we simply can't rely on a passed in filename being even vaguely decent. My preference is to call a VM routine to canonicalise the passed in string for every usage simply because you can rely (for some value of 'rely') on the OS getting it right in working out that '../../foo/bar/bim.bmp' is actually '/usr/mounts/ someOtherComputer/viaALink/mumble/frotz.jpeg'.

It also helps in case where the filenames in the image are not seen the same way the OS sees them - RISC OS uses unixy paths in Squeak because the ROS rules are so difficult to fit into the nasty habits people have of assembling strings as a path and then forcing them down the primitives. The canonicaliser call gets a chance to swap the $/ to $. before passing to the OS.

So the idea was to provide an access point to this 'service' within the core that plugins could use at need. For use in the core itself, well I guess only image loading/saving is a possible user and that may or may not be useful. For example in RISC OS you wouldn't need it for the image name to load since it would already be guaranteed to have been canonicalised *unless* you were to run from a commandline and deliberately pass in a dodgy string.

tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim When all else fails, let a = 7. If that doesn't help, then read the manual.

I believe fname here and length is technically the smalltalk object, and it's length. If you copy over from the fname it may not have a terminating 0x00.

buf[length] = 0x00

Also if you run the image as UTF-8 would copying the file name directly and assume you've get a file name that is usable by Windows, is that correct?

On 26-Mar-06, at 3:17 PM, Andreas Raab wrote:

tim Rowledge wrote:

...

...

One question about this: Do we have a way of doing that kind of file name conversion from the image itself? It strikes me as somewhat odd that a plugin needs to do that conversion given that the image can just pass a well-formed version of the string as the argument, no?

Actually, no. Until and unless we ever get some decent file handling code in the image we simply can't rely on a passed in filename being even vaguely decent. My preference is to call a VM routine to canonicalise the passed in string for every usage simply because you can rely (for some value of 'rely') on the OS getting it right in working out that '../../foo/bar/bim.bmp' is actually '/usr/mounts/someOtherComputer/viaALink/mumble/frotz.jpeg'.

All right, you guys know this better than I do (since I don't have this problem ;-) So, just for verification, the following is a legal implementation for the method, yes?

int sqGetFilenameFromString(char *buf, char *fname, int length, int b) { memcpy(buf, fname, length); return 1; }

...

tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim When all else fails, let a = 7. If that doesn't help, then read the manual.

How fitting ;-) It was just how I felt when writing the above. Are you sure this is random?

Cheers,

• Andreas

-- ======================================================================== === John M. McIntosh johnmci@smalltalkconsulting.com 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===

The mpeg plugin uses it to take the file name and to open it, also the freetype 2 plugin where it's the path to the font file.

The api then converts the given file name in VM platform character type (macroman/utf-8/latin-1) into a string that the host operating system can use to act upon using host operating system file calls.

Usage of it in the standard unix based VMs can be seen in the

Cross/sqFilePluginBasicPrims.c

sqInt sqFileOpen(SQFile *f, char* sqFileName, sqInt sqFileNameSize, sqInt writeFlag) { /* Opens the given file using the supplied sqFile structure to record its state. Fails with no side effects if f is already open. Files are always opened in binary mode; Squeak must take care of any line-end character mapping. */

char cFileName[1001];

/* don't open an already open file */ if (sqFileValid(f)) return interpreterProxy->success(false);

/* copy the file name into a null-terminated C string */ if (sqFileNameSize > 1000) { return interpreterProxy->success(false); } interpreterProxy->ioFilenamefromStringofLengthresolveAliases (cFileName, sqFileName, sqFileNameSize, true); if (writeFlag) { /* First try to open an existing file read/write: */ setFile(f, fopen(cFileName, "r+b"));

On 26-Mar-06, at 4:02 PM, Andreas Raab wrote:

John M McIntosh wrote:

...

I believe fname here and length is technically the smalltalk object, and it's length. If you copy over from the fname it may not have a terminating 0x00.

So. Where. Is. This. Used??? The comment in sq.h merely states:

"New filename converting function; used by the interpreterProxy function ioFilenamefromStringofLengthresolveAliases. Most platforms can ignore the resolveAlias boolean - it seems to only be of use by OSX but is crucial there."

and unfortunately, this doesn't give much guidance about arguments or return values. Without an example (which I have asked for, remember?) it is *impossible* to implement that function correctly based on the information I have.

Cheers,

• Andreas

-- ======================================================================== === John M. McIntosh johnmci@smalltalkconsulting.com 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===

On 26-Mar-06, at 4:35 PM, Andreas Raab wrote:

John M McIntosh wrote:

...

The mpeg plugin uses it to take the file name and to open it, also the freetype 2 plugin where it's the path to the font file.

Not in any version of the MPEG plugin that I have. Which reminds me: Did you see my message about the linker error due to the undefined reference to "sqFilenameFromStringOpen"? Could this be related?

The mpeg plugin uses sqFilenameFromStringOpen()

which I declare as

#include "sqVirtualMachine.h"

extern struct VirtualMachine * interpreterProxy;

void sqFilenameFromStringOpen(char *buffer,long fileIndex, long fileLength, int flag) { interpreterProxy->ioFilenamefromStringofLengthresolveAliases (buffer,fileIndex, fileLength, flag); }

...

The api then converts the given file name in VM platform character type (macroman/utf-8/latin-1) into a string that the host operating system can use to act upon using host operating system file calls.

Which, again, makes me want to ask why we actually burden the plugin with figuring that out instead of giving it a filename that works?

Well we need a primitive that would translate the image file string to the host api file name. ioFilenamefromStringofLengthresolveAliases comes to mind.

Then pass in the translated results, however now it's an issue of backward/forward compatibility.

...

Usage of it in the standard unix based VMs can be seen in the Cross/sqFilePluginBasicPrims.c /* copy the file name into a null-terminated C string */ if (sqFileNameSize > 1000) { return interpreterProxy->success(false); } interpreterProxy->ioFilenamefromStringofLengthresolveAliases (cFileName, sqFileName, sqFileNameSize, true);

I see. If the other usages are the same then the one thing that's missing from my implementation is a terminating zero character, yes?

Yes

And like David mentioned, the above is prone to buffer overflow attacks whenever the VMs encoding is larger than the image's (which is easily possible if you consider, e.g., latin1 -> utf8 conversion).

Ya, likely we should pass in the max size of the target buffer.

Cheers,

• Andreas

-- ======================================================================== === John M. McIntosh johnmci@smalltalkconsulting.com 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===

Hello,

Well, my point is that if we expose that primitive to the image, we will never have the need to do it from a plugin because you can always pass a valid file name into it. It's really strange to me that the VM does translation for stuff which is (supposedly) handled in the image already. I could understand this if we would shoot for a file/directory abstraction inside the image but the way it's right now we're hiding stuff from the image that the image supposedly is responsible for (platform file name handling) and that's Just Wrong (tm).

Since Squeak3.8, FilePath class is supposed to do the conversion. I wonder if how ioFilename:fromString:ofLength:resolveAliases: interacts with FilePath...

-- Yoshiki

John,

It may be done poorly. But the idea is to switch the encoding in the image by providing the right fileNameConverter.

I'm not sure what you mean by "in image encoding assumes latin-1". If the file browser doesn't show the right file name, the right fix is to change the fileNameConverter, not changing the VM.

-- Yoshiki

At Wed, 29 Mar 2006 21:35:47 -0800, John M McIntosh wrote:

Ah, poorly.

From what I've seen the in image encoding assumes latin-1. It gets macroman from the VM and mangles, demangles that ok, however displays the characters then wrong in the file browser. If you switch the VM to use latin-1 then characters are displayed correctly, however it is possible to create file names that contains characters that UTF-8 allows but has not latin-1 mapping, thus the internal VM mapping from that character to latin-1 fails.

Keeping the VM in UTF-8 seems to work, of course FilePath mangles that as latin-1 and the names are not displayed correctly.

On 29-Mar-06, at 6:42 PM, Yoshiki Ohshima wrote:

...

Hello,

...

Well, my point is that if we expose that primitive to the image, we will never have the need to do it from a plugin because you can always pass a valid file name into it. It's really strange to me that the VM does translation for stuff which is (supposedly) handled in the image already. I could understand this if we would shoot for a file/ directory abstraction inside the image but the way it's right now we're hiding stuff from the image that the image supposedly is responsible for (platform file name handling) and that's Just Wrong (tm).

Since Squeak3.8, FilePath class is supposed to do the conversion. I wonder if how ioFilename:fromString:ofLength:resolveAliases: interacts with FilePath...

-- Yoshiki

--

=== John M. McIntosh johnmci@smalltalkconsulting.com 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===

John,

Well the VM is given a info.plist entry or cmd line that defines what encoding to use. This information needs to be gotten from the VM in order to set the proper filenameConverter. My understanding was that the default for the mac carbon vm was MacRoman, but the fileNameConverter used in 3.8? is latin-1. If I choose latin-1 there are characters in the utf-8 character set allowed in os-x that can't be translated to latin-1 so latin-1 technically is not a viable choice. I understand UTF-8 is viable, but somehow the image should configure that from the Vm without manual intervention.

Note that the fileNameConverter to use is decided when the image is started. It can check the platform, vm version, etc., etc. to decide which one to use. An example is shown JapaneseEnvironment class>>systemConverterClass. In the other words, the image stuff can be configured.

-- Yoshiki

The image defaults to Latin1Environment on Mac, but John's VM did not support Latin1 until very recently. It does support UTF-8, however, there is no UTF8Environment that could be set as default.

A hack that I used was to manually set the filename converter class to UTF-8 after startup, but that's cumbersome.

- Bert -

Am 02.04.2006 um 04:38 schrieb Yoshiki Ohshima:

John,

It may be done poorly. But the idea is to switch the encoding in the image by providing the right fileNameConverter.

I'm not sure what you mean by "in image encoding assumes latin-1". If the file browser doesn't show the right file name, the right fix is to change the fileNameConverter, not changing the VM.

-- Yoshiki

At Wed, 29 Mar 2006 21:35:47 -0800, John M McIntosh wrote:

...

Ah, poorly.

From what I've seen the in image encoding assumes latin-1. It gets macroman from the VM and mangles, demangles that ok, however displays the characters then wrong in the file browser. If you switch the VM to use latin-1 then characters are displayed correctly, however it is possible to create file names that contains characters that UTF-8 allows but has not latin-1 mapping, thus the internal VM mapping from that character to latin-1 fails.

Keeping the VM in UTF-8 seems to work, of course FilePath mangles that as latin-1 and the names are not displayed correctly.

On 29-Mar-06, at 6:42 PM, Yoshiki Ohshima wrote:

...

Hello,

...

Well, my point is that if we expose that primitive to the image, we will never have the need to do it from a plugin because you can always pass a valid file name into it. It's really strange to me that the VM does translation for stuff which is (supposedly) handled in the image already. I could understand this if we would shoot for a file/ directory abstraction inside the image but the way it's right now we're hiding stuff from the image that the image supposedly is responsible for (platform file name handling) and that's Just Wrong (tm).

Since Squeak3.8, FilePath class is supposed to do the conversion. I wonder if how ioFilename:fromString:ofLength:resolveAliases: interacts with FilePath...

-- Yoshiki

--

===

John M. McIntosh johnmci@smalltalkconsulting.com 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http:// www.smalltalkconsulting.com ===================================================================== === ===