Hi Chris,
On Fri, Sep 11, 2015 at 9:01 AM, Chris Cunnington brasspen@gmail.com wrote:
I had the idea that a plugin from one VM could be dragged into another VM's directory and that it could be used just by starting up the image. I've done a little experimenting and it seems more of an idea of a reality. The functionality is there, but VM developers over the years have not seen this as a priority. Typically plugins and their VMs are compiled together. Or a person adds one by recompiling a VM compilation rig they already have.
The problems I see with this are
a) a plugin compiled for Spur may not work with V3 or vice verse. The issue is the header size of an object. Some plugins, but not all, use this define, and the header sizes between Spur and V3 are incompatible.
b) 32-bit plugins won't work with 64-bit VMs, and 64-bit plugins won't work with 32-bit VMs. Period.
Now there are platform-level packaging technologies, such as fat binaries on Mac OS X, that allow one to construct plugins that contain more than one binary. But this is a lot of work to build and maintain.
So personally I wouldn't put much effort into this level of drag-and-drop compatibility. Instead I'd put energy into good error messages so that when plugins don't work the user can find out why, and that when the wrong kind of plugin is used the VM doesn't just stumble along, maybe producing incorrect results, but instead puts the plugin out with a comprehensible complaint.
Does this make sense? I know its a downer, but what you propose is, IMO, not affordable given our resources.
And I must say, *this is a VM-DEV discussion, not a general purpose Squeak discussion*, yes?
As I had a rig for the Interpreter VM I decided to compile a plugin and see
if I could drag it around to other VM directories for use. Most times it didn't work.
I compiled a TheUniversalAnswer plugin which returns 42. I moved it from a 4.14.1-3430 VM to a 4.14.1-3414 VM. I could get it to work if I did not use the squeak.sh start script. That is, I dragged so.vm-sound-null, so.TheUniversalAnswer and so.vm-display-X11 into the same directory as the VM binary. That worked.
The only way to see the external plugins is with #listLoadedModules. But, irritatingly, modules are loaded as needed, so once you have proof of using the primitive from the plugin, yea, it will appear as a result of that Smalltalk listLoadedModules message. So, it's not that useful.
I dragged so.TheUniversalAnswer around to other VMs such as 4.0.3-2202 and 4.13.10-3268 without success. And the unload selectors #forgetModule: and unloadModule:, which both use primitive 571 don't appear to work on my Ubuntu 15.04.
So, I don't know if a community of plugins passed around is in the offing. What I have learned is that many of the answers are in sqUnixExternalPrims.c. The issues are really where does the VM look for primitives and what does it do when it finds one. It appears to me, after reading both the non-Cog and Cog versions of that file, that this an interesting area poorly documented that VM developers alter in haste to get on to something else.
It's pretty important stuff, though. Do you start the VM with the binary on the path? Do you use squeak.sh? Where is the VM looking for stuff. Where will FFI look for stuff. And so on.
Chris
Eliot,
I agree. Indeed, even with an interpreter VM it has long been the case that a plugin compiled as 64 bit will not work on a VM compiled for 64 bit and vice versa. With the number of variations of 32/64, Spur/V3, and so forth, I don't think that this is a problem worth worrying about. Better to just be a aware that plugins are not readily interchangeable in those cases.
Dave
Hi Chris,
On Fri, Sep 11, 2015 at 9:01 AM, Chris Cunnington brasspen@gmail.com wrote:
I had the idea that a plugin from one VM could be dragged into another VM's directory and that it could be used just by starting up the image. I've done a little experimenting and it seems more of an idea of a reality. The functionality is there, but VM developers over the years have not seen this as a priority. Typically plugins and their VMs are compiled together. Or a person adds one by recompiling a VM compilation rig they already have.
The problems I see with this are
a) a plugin compiled for Spur may not work with V3 or vice verse. The issue is the header size of an object. Some plugins, but not all, use this define, and the header sizes between Spur and V3 are incompatible.
b) 32-bit plugins won't work with 64-bit VMs, and 64-bit plugins won't work with 32-bit VMs. Period.
Now there are platform-level packaging technologies, such as fat binaries on Mac OS X, that allow one to construct plugins that contain more than one binary. But this is a lot of work to build and maintain.
So personally I wouldn't put much effort into this level of drag-and-drop compatibility. Instead I'd put energy into good error messages so that when plugins don't work the user can find out why, and that when the wrong kind of plugin is used the VM doesn't just stumble along, maybe producing incorrect results, but instead puts the plugin out with a comprehensible complaint.
Does this make sense? I know its a downer, but what you propose is, IMO, not affordable given our resources.
And I must say, *this is a VM-DEV discussion, not a general purpose Squeak discussion*, yes?
As I had a rig for the Interpreter VM I decided to compile a plugin and see
if I could drag it around to other VM directories for use. Most times it didn't work.
I compiled a TheUniversalAnswer plugin which returns 42. I moved it from a 4.14.1-3430 VM to a 4.14.1-3414 VM. I could get it to work if I did not use the squeak.sh start script. That is, I dragged so.vm-sound-null, so.TheUniversalAnswer and so.vm-display-X11 into the same directory as the VM binary. That worked.
The only way to see the external plugins is with #listLoadedModules. But, irritatingly, modules are loaded as needed, so once you have proof of using the primitive from the plugin, yea, it will appear as a result of that Smalltalk listLoadedModules message. So, it's not that useful.
I dragged so.TheUniversalAnswer around to other VMs such as 4.0.3-2202 and 4.13.10-3268 without success. And the unload selectors #forgetModule: and unloadModule:, which both use primitive 571 don't appear to work on my Ubuntu 15.04.
So, I don't know if a community of plugins passed around is in the offing. What I have learned is that many of the answers are in sqUnixExternalPrims.c. The issues are really where does the VM look for primitives and what does it do when it finds one. It appears to me, after reading both the non-Cog and Cog versions of that file, that this an interesting area poorly documented that VM developers alter in haste to get on to something else.
It's pretty important stuff, though. Do you start the VM with the binary on the path? Do you use squeak.sh? Where is the VM looking for stuff. Where will FFI look for stuff. And so on.
Chris
-- _,,,^..^,,,_ best, Eliot
On 11-09-2015, at 12:00 PM, Eliot Miranda eliot.miranda@gmail.com wrote:
a) a plugin compiled for Spur may not work with V3 or vice verse. The issue is the header size of an object. Some plugins, but not all, use this define, and the header sizes between Spur and V3 are incompatible.
b) 32-bit plugins won't work with 64-bit VMs, and 64-bit plugins won't work with 32-bit VMs. Period.
At the very least we should make sure the version checking that was designed specifically to help handle this is actually getting used properly. That ought to prevent crashes and help with providing some decent error messages.
Now there are platform-level packaging technologies, such as fat binaries on Mac OS X, that allow one to construct plugins that contain more than one binary. But this is a lot of work to build and maintain.
I suspect we could avoid that by using the versioning stuff to indicate which plugin file(s) the user/system fetches?
As I (re)discovered this week, the unix vm plugin/library loading is amazingly convoluted and looks in more directories than I think my Pi actually has. And it uses the name you give it prepended with ‘lib’ and appended with ‘.so’ and ‘dylib’, so searching for a plugin/lib involves an awful lot of checking and file testing. Looking in the platforms/unix/vm/sqUnixExternalPrims.c you can see what is being done. Note that ther eare two completely different approaches in that code and the Cog VM is using (so far as I can tell) the NOT USE_SIMPLIFIED_PLUGIN_LOGIC branch. The (somewhat old) copy of the plain interpreter VM equivalent I have around DOES use the simplified logic. We should probably clean that up...
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Fractured Idiom:- AMICUS PURIAE - Platonic friend
On 11-09-2015, at 12:00 PM, Eliot Miranda eliot.miranda@gmail.com wrote:
a) a plugin compiled for Spur may not work with V3 or vice verse. The issue is the header size of an object. Some plugins, but not all, use this define, and the header sizes between Spur and V3 are incompatible.
b) 32-bit plugins won't work with 64-bit VMs, and 64-bit plugins won't work with 32-bit VMs. Period.
At the very least we should make sure the version checking that was designed specifically to help handle this is actually getting used properly.
That's a good point, although it probably will not help with a many of these cases. For example, with an interpreter VM the various permutations of 32/64 bit host and 32/64 bit object memory are all compiled from a single identical code base, but the plugins cannot be mixed among them. On the other hand, it might be helpful for handling cases such as someone trying to run a Cog plugin with a Spur VM, or trying to mix 32 bit plugins with a 64 bit Spur VM.
That ought to prevent crashes and help with providing some decent error messages.
In the case of mixing 32/64 bit compiled modules, the OS loader will not let this happen, so the only issue is that the plugin (quite rightly) does not load. But there might be other mixes of VM and plugin that could result in a crash.
Dave
Now there are platform-level packaging technologies, such as fat binaries on Mac OS X, that allow one to construct plugins that contain more than one binary. But this is a lot of work to build and maintain.
I suspect we could avoid that by using the versioning stuff to indicate which plugin file(s) the user/system fetches?
As I (re)discovered this week, the unix vm plugin/library loading is amazingly convoluted and looks in more directories than I think my Pi actually has. And it uses the name you give it prepended with lib and appended with .so and dylib, so searching for a plugin/lib involves an awful lot of checking and file testing. Looking in the platforms/unix/vm/sqUnixExternalPrims.c you can see what is being done. Note that ther eare two completely different approaches in that code and the Cog VM is using (so far as I can tell) the NOT USE_SIMPLIFIED_PLUGIN_LOGIC branch. The (somewhat old) copy of the plain interpreter VM equivalent I have around DOES use the simplified logic. We should probably clean that up...
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Fractured Idiom:- AMICUS PURIAE - Platonic friend
On 11-09-2015, at 1:15 PM, David T. Lewis lewis@mail.msen.com wrote:
At the very least we should make sure the version checking that was designed specifically to help handle this is actually getting used properly.
That's a good point, although it probably will not help with a many of these cases. For example, with an interpreter VM the various permutations of 32/64 bit host and 32/64 bit object memory are all compiled from a single identical code base, but the plugins cannot be mixed among them.
Sounds like maybe modifying the version system to include some signifier for that might help?
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: RG: Read inter-record Gap
On 11-09-2015, at 1:15 PM, David T. Lewis lewis@mail.msen.com wrote:
At the very least we should make sure the version checking that was designed specifically to help handle this is actually getting used properly.
That's a good point, although it probably will not help with a many of these cases. For example, with an interpreter VM the various permutations of 32/64 bit host and 32/64 bit object memory are all compiled from a single identical code base, but the plugins cannot be mixed among them.
Sounds like maybe modifying the version system to include some signifier for that might help?
It would be a good thing on general principles to make sure the versioning system is sane (I have not paid attention to it). But it can't handle the case of mixing 32 and 64 bit compiled modules, because the modules are not going to load in the first place (the OS will see to that). I might be the only person around who regularly uses the V3 64 bit object memory, so you don't need to worry about protecting me from mixing the plugins, I already know not to do that ;-)
It might be helpful for protecting against the case of mixing up 32-bit compiled Cog and Spur modules.
Dave
On Sat, Sep 12, 2015 at 5:16 AM, David T. Lewis lewis@mail.msen.com wrote:
On 11-09-2015, at 1:15 PM, David T. Lewis lewis@mail.msen.com wrote:
At the very least we should make sure the version checking that was designed specifically to help handle this is actually getting used properly.
That's a good point, although it probably will not help with a many of these cases. For example, with an interpreter VM the various permutations of 32/64 bit host and 32/64 bit object memory are all compiled from a single identical code base, but the plugins cannot be mixed among them.
Sounds like maybe modifying the version system to include some signifier for that might help?
It would be a good thing on general principles to make sure the versioning system is sane (I have not paid attention to it). But it can't handle the case of mixing 32 and 64 bit compiled modules, because the modules are not going to load in the first place (the OS will see to that). I might be the only person around who regularly uses the V3 64 bit object memory, so you don't need to worry about protecting me from mixing the plugins, I already know not to do that ;-)
It might be helpful for protecting against the case of mixing up 32-bit compiled Cog and Spur modules.
Just a random idea that may not be practical or worth the effort, you might have a system where a plugin can be dragged onto the Image desktop, and something in the binary header not to hard to analyse could report whether the plugin is compatible and suggest where to put it. cheers -ben
On 2015-09-11 3:18 PM, tim Rowledge wrote:
On 11-09-2015, at 12:00 PM, Eliot Miranda eliot.miranda@gmail.com wrote:
a) a plugin compiled for Spur may not work with V3 or vice verse. The issue is the header size of an object. Some plugins, but not all, use this define, and the header sizes between Spur and V3 are incompatible.
b) 32-bit plugins won't work with 64-bit VMs, and 64-bit plugins won't work with 32-bit VMs. Period.
At the very least we should make sure the version checking that was designed specifically to help handle this is actually getting used properly. That ought to prevent crashes and help with providing some decent error messages.
Now there are platform-level packaging technologies, such as fat binaries on Mac OS X, that allow one to construct plugins that contain more than one binary. But this is a lot of work to build and maintain.
I suspect we could avoid that by using the versioning stuff to indicate which plugin file(s) the user/system fetches?
As I (re)discovered this week, the unix vm plugin/library loading is amazingly convoluted and looks in more directories than I think my Pi actually has. And it uses the name you give it prepended with ‘lib’ and appended with ‘.so’ and ‘dylib’, so searching for a plugin/lib involves an awful lot of checking and file testing. Looking in the platforms/unix/vm/sqUnixExternalPrims.c you can see what is being done. Note that ther eare two completely different approaches in that code and the Cog VM is using (so far as I can tell) the NOT USE_SIMPLIFIED_PLUGIN_LOGIC branch. The (somewhat old) copy of the plain interpreter VM equivalent I have around DOES use the simplified logic. We should probably clean that up...
Oh my. Both the Cog and Interpreter sqUnixExternalPrims.c have two implementations of ioLoadModule(). I think Cog ends up leaning on tryLoading() while the Interpreter leans on tryLoadModule(). It's appalling. Just count the number of uses of dlopen(). I'm no C programmer and following this stuff through step by step with gdb was needless torture. It's the reason my 32-bit Cog/Stack builds fail on Ubuntu 15.04. **shivers**
I think the reason is that nobody wants to offend their forebearers. The three of you attend Squeak Board meetings. Come to some kind of agreement on what can be ripped out and then send Ian an nice note.
What's most revolting is there are about five systems of tests looking for every possible combination (i.e. so.vm-display-X11, vm-display-X11.so, vm-display-X11.dylib, etc.) and they all fail. Only the last one works or something like that. **shivers like Bart Simpson**
Chris
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Fractured Idiom:- AMICUS PURIAE - Platonic friend
On Fri, Sep 11, 2015 at 07:29:11PM -0400, Chris Cunnington wrote:
On 2015-09-11 3:18 PM, tim Rowledge wrote:
As I (re)discovered this week, the unix vm plugin/library loading is amazingly convoluted and looks in more directories than I think my Pi actually has. And it uses the name you give it prepended with ?lib? and appended with ?.so? and ?dylib?, so searching for a plugin/lib involves an awful lot of checking and file testing. Looking in the platforms/unix/vm/sqUnixExternalPrims.c you can see what is being done. Note that ther eare two completely different approaches in that code and the Cog VM is using (so far as I can tell) the NOT USE_SIMPLIFIED_PLUGIN_LOGIC branch. The (somewhat old) copy of the plain interpreter VM equivalent I have around DOES use the simplified logic. We should probably clean that up...
Oh my. Both the Cog and Interpreter sqUnixExternalPrims.c have two implementations of ioLoadModule(). I think Cog ends up leaning on tryLoading() while the Interpreter leans on tryLoadModule(). It's appalling. Just count the number of uses of dlopen(). I'm no C programmer and following this stuff through step by step with gdb was needless torture. It's the reason my 32-bit Cog/Stack builds fail on Ubuntu 15.04. **shivers**
I think the reason is that nobody wants to offend their forebearers. The three of you attend Squeak Board meetings. Come to some kind of agreement on what can be ripped out and then send Ian an nice note.
:-)
I am quite confident that Ian would not be unhappy if someone were to send him a nice note. That does not happen often enough. Ian is the author of the Unix VM platform code, and a number of other things that the Squeak/Pharo community rely on every day. I think that Andreas Raab was also responsible for developing the module lookup strategy.
As far as I am aware, the module search logic in SVN trunk would be the latest and presumably best version of Ian's implementation. The oscog branch was forked at an earlier time, and may have differences that reflect an earlier version of Ian's work, at least with respect to module lookup logic.
Unless there is some compelling reason to look elsewhere, the latest version of Ian's work is in SVN trunk, and that should serve as the reference implementation with respect to the module lookup strategy for plugin modules and VM loadable modules (another of Ian's related contributions).
Dave
Oh my. Both the Cog and Interpreter sqUnixExternalPrims.c have two implementations of ioLoadModule(). I think Cog ends up leaning on tryLoading() while the Interpreter leans on tryLoadModule(). It's appalling. Just count the number of uses of dlopen(). I'm no C programmer and following this stuff through step by step with gdb was needless torture. It's the reason my 32-bit Cog/Stack builds fail on Ubuntu 15.04. **shivers**
What's most revolting is there are about five systems of tests looking for every possible combination (i.e. so.vm-display-X11, vm-display-X11.so, vm-display-X11.dylib, etc.) and they all fail. Only the last one works or something like that. **shivers like Bart Simpson**
I think the reason is that nobody wants to offend their forebearers. The [four] of you attend Squeak Board meetings. Come to some kind of agreement on what can be ripped out and then send Ian an nice note.
Well said. Yes, let's.
-C
-- Craig Latta netjam.org +31 6 2757 7177 (SMS ok) + 1 415 287 3547 (no SMS)
On 2015-09-11 3:00 PM, Eliot Miranda wrote:
Hi Chris,
On Fri, Sep 11, 2015 at 9:01 AM, Chris Cunnington <brasspen@gmail.com mailto:brasspen@gmail.com> wrote:
I had the idea that a plugin from one VM could be dragged into another VM's directory and that it could be used just by starting up the image. I've done a little experimenting and it seems more of an idea of a reality. The functionality is there, but VM developers over the years have not seen this as a priority. Typically plugins and their VMs are compiled together. Or a person adds one by recompiling a VM compilation rig they already have.The problems I see with this are
a) a plugin compiled for Spur may not work with V3 or vice verse. The issue is the header size of an object. Some plugins, but not all, use this define, and the header sizes between Spur and V3 are incompatible.
b) 32-bit plugins won't work with 64-bit VMs, and 64-bit plugins won't work with 32-bit VMs. Period.
Now there are platform-level packaging technologies, such as fat binaries on Mac OS X, that allow one to construct plugins that contain more than one binary. But this is a lot of work to build and maintain.
So personally I wouldn't put much effort into this level of drag-and-drop compatibility. Instead I'd put energy into good error messages so that when plugins don't work the user can find out why, and that when the wrong kind of plugin is used the VM doesn't just stumble along, maybe producing incorrect results, but instead puts the plugin out with a comprehensible complaint.
I looked up some old Igor pdfs and I think I see what you mean. #primtitiveFailFor: and #initializePrimitiveErrorCodes. Yea, that looks like a better focus of energy.
Does this make sense? I know its a downer, but what you propose is, IMO, not affordable given our resources.
Agreed.
And I must say, *this is a VM-DEV discussion, not a general purpose Squeak discussion*, yes?
Yes.
Chris
As I had a rig for the Interpreter VM I decided to compile a plugin and see if I could drag it around to other VM directories for use. Most times it didn't work. I compiled a TheUniversalAnswer plugin which returns 42. I moved it from a 4.14.1-3430 VM to a 4.14.1-3414 VM. I could get it to work if I did not use the squeak.sh start script. That is, I dragged so.vm-sound-null, so.TheUniversalAnswer and so.vm-display-X11 into the same directory as the VM binary. That worked. The only way to see the external plugins is with #listLoadedModules. But, irritatingly, modules are loaded as needed, so once you have proof of using the primitive from the plugin, yea, it will appear as a result of that Smalltalk listLoadedModules message. So, it's not that useful. I dragged so.TheUniversalAnswer around to other VMs such as 4.0.3-2202 and 4.13.10-3268 without success. And the unload selectors #forgetModule: and unloadModule:, which both use primitive 571 don't appear to work on my Ubuntu 15.04. So, I don't know if a community of plugins passed around is in the offing. What I have learned is that many of the answers are in sqUnixExternalPrims.c. The issues are really where does the VM look for primitives and what does it do when it finds one. It appears to me, after reading both the non-Cog and Cog versions of that file, that this an interesting area poorly documented that VM developers alter in haste to get on to something else. It's pretty important stuff, though. Do you start the VM with the binary on the path? Do you use squeak.sh? Where is the VM looking for stuff. Where will FFI look for stuff. And so on. Chris-- _,,,^..^,,,_ best, Eliot
vm-dev@lists.squeakfoundation.org
-
Ben Coman -
Chris Cunnington -
Craig Latta -
David T. Lewis -
Eliot Miranda -
tim Rowledge