I was pretty unhappy with some places in VM, where it loads a function from plugin by calling ioLoadFunction... Also, even more painful to see when one plugin wants to call another plugin function.
IMO things would be much better , if we formalize these things in VM. An idea is simple and easy to implement:
We need only few functions implemented in VM:
sqInt makeAtom(char * name); "registers new atom or returns id of already existing one"
sqInt registerService (sqInt atom, void * service); "associate a value with service id" sqInt unregisterService(sqInt atom); "clear association with service id (make value=0)"
And finally,
void * getService(sqInt atom);
Now, plugins first, should declare the atoms they would want to use or provide. This can be done once at plugin initialization stage, for instance:
static sqInt bitBlitAtom = makeAtom('ioBitBlt');
Now, after acquiring atom, plugin can register a service fn under this atom:
registerService(bitBlitAtom, (void*) bitBlt); and upon unloading unregisterService(bitBlitAtom);
now, any other plugin can do:
bitBltFn = getService(bitBlitAtom);
if (bitBltFn) { bitBltFn( a, b,c blabla); }
VM maintains a simple list of atom values and list of atom names and their numeric correspondence. A getService(bitBlitAtom) is very fast, because its simple access by index:
getService(sqInt atom) { if (atom>=0 && atom<atomsCount) return atoms[atom]; return 0; }
Igor Stasenko wrote:
I was pretty unhappy with some places in VM, where it loads a function from plugin by calling ioLoadFunction... Also, even more painful to see when one plugin wants to call another plugin function.
Why would it be advantageous to write:
static sqInt bitBlitAtom = makeAtom('ioBitBlt'); registerService(bitBlitAtom, (void*) bitBlt); bitBltFn = getService(bitBlitAtom);
instead of using
bitBltFn = ioLoadFunctionFrom("ioBitBlt", "BitBltPlugin");
Is there any reason for making things even more lengthy than they are already?
Cheers, - Andreas
IMO things would be much better , if we formalize these things in VM. An idea is simple and easy to implement:
We need only few functions implemented in VM:
sqInt makeAtom(char * name); "registers new atom or returns id of already existing one"
sqInt registerService (sqInt atom, void * service); "associate a value with service id" sqInt unregisterService(sqInt atom); "clear association with service id (make value=0)"
And finally,
void * getService(sqInt atom);
Now, plugins first, should declare the atoms they would want to use or provide. This can be done once at plugin initialization stage, for instance:
static sqInt bitBlitAtom = makeAtom('ioBitBlt');
Now, after acquiring atom, plugin can register a service fn under this atom:
registerService(bitBlitAtom, (void*) bitBlt); and upon unloading unregisterService(bitBlitAtom);
now, any other plugin can do:
bitBltFn = getService(bitBlitAtom);
if (bitBltFn) { bitBltFn( a, b,c blabla); }
VM maintains a simple list of atom values and list of atom names and their numeric correspondence. A getService(bitBlitAtom) is very fast, because its simple access by index:
getService(sqInt atom) { if (atom>=0 && atom<atomsCount) return atoms[atom]; return 0; }
2008/11/21 Andreas Raab andreas.raab@gmx.de:
Igor Stasenko wrote:
I was pretty unhappy with some places in VM, where it loads a function from plugin by calling ioLoadFunction... Also, even more painful to see when one plugin wants to call another plugin function.
Why would it be advantageous to write:
static sqInt bitBlitAtom = makeAtom('ioBitBlt'); registerService(bitBlitAtom, (void*) bitBlt); bitBltFn = getService(bitBlitAtom);
instead of using
bitBltFn = ioLoadFunctionFrom("ioBitBlt", "BitBltPlugin");
Is there any reason for making things even more lengthy than they are already?
First, you are statically associating functionality with specific module ("ioBitBlt" in your example), while using atoms i don't care which module may set it , i care only about specific functionality. Second, with ioLoadFunctionFrom you can retrieve function, not arbitrary value(s). Third - you can change atom value dynamically , while ioLoadFunctionFrom doomed to return same value all the time - null if no module found or no such function in module , or function address on success. Fourth, there are many cases where i wouldn't want to expose functions of my module directly (by exporting them), only indirectly.
Cheers,
- Andreas
IMO things would be much better , if we formalize these things in VM. An idea is simple and easy to implement:
We need only few functions implemented in VM:
sqInt makeAtom(char * name); "registers new atom or returns id of already existing one"
sqInt registerService (sqInt atom, void * service); "associate a value with service id" sqInt unregisterService(sqInt atom); "clear association with service id (make value=0)"
And finally,
void * getService(sqInt atom);
Now, plugins first, should declare the atoms they would want to use or provide. This can be done once at plugin initialization stage, for instance:
static sqInt bitBlitAtom = makeAtom('ioBitBlt');
Now, after acquiring atom, plugin can register a service fn under this atom:
registerService(bitBlitAtom, (void*) bitBlt); and upon unloading unregisterService(bitBlitAtom);
now, any other plugin can do:
bitBltFn = getService(bitBlitAtom);
if (bitBltFn) { bitBltFn( a, b,c blabla); }
VM maintains a simple list of atom values and list of atom names and their numeric correspondence. A getService(bitBlitAtom) is very fast, because its simple access by index:
getService(sqInt atom) { if (atom>=0 && atom<atomsCount) return atoms[atom]; return 0; }
Igor Stasenko wrote:
Why would it be advantageous to write:
static sqInt bitBlitAtom = makeAtom('ioBitBlt'); registerService(bitBlitAtom, (void*) bitBlt); bitBltFn = getService(bitBlitAtom);
instead of using
bitBltFn = ioLoadFunctionFrom("ioBitBlt", "BitBltPlugin");
Is there any reason for making things even more lengthy than they are already?
First, you are statically associating functionality with specific module ("ioBitBlt" in your example), while using atoms i don't care which module may set it , i care only about specific functionality.
I see. This wasn't clear in your example. What use cases do you have in mind? I have only once needed something like this which was during an interim period where we supported two BitBlt variants and I wanted to be able to use either one from Balloon.
Also, keep in mind that the unstructured namespace you are proposing can easily lead to conflicts - one of the reasons why ioLoadFunctionFrom is explicit about providing the plugin name is because of all of these "primitiveVersion" implementations out there. If you use the flat namespace throughout the system you'll have to have a need to disambiguate somehow.
Second, with ioLoadFunctionFrom you can retrieve function, not arbitrary value(s).
ioLoadFunctionFrom retrieves pointers and they can of course point to values. For example, the Windows VM exports references to the main window ("stWindow") and message hooks. Check out <platform>/vm/sq<Plat>Exports.c for examples of platform specific exports.
Third - you can change atom value dynamically , while ioLoadFunctionFrom doomed to return same value all the time - null if no module found or no such function in module , or function address on success.
Well, of course for changing values, you export functions ;-) Seems like the most obvious reason for exporting a function instead of a value. Though either way will work.
Fourth, there are many cases where i wouldn't want to expose functions of my module directly (by exporting them), only indirectly.
Again, I'd be interested in knowing more about your use cases. It is certainly doable to do this in the VM but we should be careful about adding duplicate functionality.
Cheers, - Andreas
2008/11/22 Andreas Raab andreas.raab@gmx.de:
Igor Stasenko wrote:
Why would it be advantageous to write:
static sqInt bitBlitAtom = makeAtom('ioBitBlt'); registerService(bitBlitAtom, (void*) bitBlt); bitBltFn = getService(bitBlitAtom);
instead of using
bitBltFn = ioLoadFunctionFrom("ioBitBlt", "BitBltPlugin");
Is there any reason for making things even more lengthy than they are already?
First, you are statically associating functionality with specific module ("ioBitBlt" in your example), while using atoms i don't care which module may set it , i care only about specific functionality.
I see. This wasn't clear in your example. What use cases do you have in mind? I have only once needed something like this which was during an interim period where we supported two BitBlt variants and I wanted to be able to use either one from Balloon.
Also, keep in mind that the unstructured namespace you are proposing can easily lead to conflicts - one of the reasons why ioLoadFunctionFrom is explicit about providing the plugin name is because of all of these "primitiveVersion" implementations out there. If you use the flat namespace throughout the system you'll have to have a need to disambiguate somehow.
The main reason why to use the flat namespace is to allow sharing state identified by name between modules without exact knowledge where this state comes from. This, of course implies some contracts about meaning of symbols , e.g. "bitblt" is a function pointer which takes N arguments and so on (but hey, you implying the same contract anyways, when calling ioLoadFunctionFrom("ioBitBlt", "BitBltPlugin") - just with one exception, that you have to explicitly specify the source module where it comes from). This common namespace can act as meeting point , provided by VM, where plugins can share data without knowing exact source of it. As for ambiguous names: we have not much plugins to care about this problem so far :) In any case - this problem can be addressed in same way as currently in squeak e.g. define 'abcValue' instead of 'Value'. Or define 'mymodule.myname' .. any form which you may prefer. But in latter case, 'mymodule.myname' is nothing more better than ioLoadFunctionFrom("myname", "mymodule") and simply indicates that you abusing the global namespace by defining too specific values.
And finally, if we care so much about ambiguousness , one can always register symbols like 'myModule.version' so others could check if version is ok.
Second, with ioLoadFunctionFrom you can retrieve function, not arbitrary value(s).
ioLoadFunctionFrom retrieves pointers and they can of course point to values. For example, the Windows VM exports references to the main window ("stWindow") and message hooks. Check out <platform>/vm/sq<Plat>Exports.c for examples of platform specific exports.
Third - you can change atom value dynamically , while ioLoadFunctionFrom doomed to return same value all the time - null if no module found or no such function in module , or function address on success.
Well, of course for changing values, you export functions ;-) Seems like the most obvious reason for exporting a function instead of a value. Though either way will work.
Fourth, there are many cases where i wouldn't want to expose functions of my module directly (by exporting them), only indirectly.
Again, I'd be interested in knowing more about your use cases. It is certainly doable to do this in the VM but we should be careful about adding duplicate functionality.
One case, as you pointed out, is when you having multiple versions of same plugin and want to swap them on the fly.
Smalltalk unloadModule: 'OldBlt'. Smalltalk loadModule: 'NewBlt'.
Another case, is when you want to overlook some activity by installing a proxy which will log all calls to some functions.
Smalltalk loadModule: 'SecurityLogger'. ... do some testing/debugging etc .. Smalltalk unloadModule: 'SecurityLogger'
Third case:
Smalltalk unloadModule: 'UnwantedStuff'.
if there another hideous plugin (or VM) which using ioLoadFunctionFrom(), it would be very hard to prevent from loading it again :)
Another one, as variant of first case (OldBlt/NewBlt), suppose you have module 'fooBar' which using functionality of some 'defaultFoo' module. But as well, you have a 'highEndFoo' module, loaded as well which having same interface as defaultFoo, but slightly different implementation:
Smalltalk lowerCPUUsage: bool bool ifTrue: [ HighEndFoo deactivate ] ifFalse: [ HighEndFoo reactivate ]
Note, that here, #deactivate wont lead to unloading a module. It stays in memory, ready to be used later, but simply rerouting things back to 'defaultFoo'.
I'm not saying that there's much pressing need in such use cases. But these 3 functions will give us more flexibility , security and modularity.
To be fair , i'd make all plugins to export only single function, like setInterpreter(), which then tells VM what things it wants to expose. Currently, all plugins exposing functions and primitives in static manner - and have no control at run time what to expose or not. But that's another story :)
Cheers,
- Andreas
If we follow further this road, potentially, we could associate each primitive with atom. E.g. instead of holding direct function pointers in external primitive table, we register an atom/symbol with moduleName.primitiveName and put pointer there.
Now new module can selectively replace existing primitive with new one, even #primitiveAdd, with own , better faster implementation, if it makes any sense :)
Igor Stasenko wrote:
The main reason why to use the flat namespace is to allow sharing state identified by name between modules without exact knowledge where this state comes from. This, of course implies some contracts about meaning of symbols , e.g. "bitblt" is a function pointer which takes N arguments and so on (but hey, you implying the same contract anyways, when calling ioLoadFunctionFrom("ioBitBlt", "BitBltPlugin") - just with one exception, that you have to explicitly specify the source module where it comes from).
Well, what's interesting about this is that it becomes much more of an issue if this mechanism is widely used to share function pointers. Right now, 99% of the entry points into plugins are primitives which use the Smalltalk stack and do their own argument checking. There is only a tiny number of functions that is used directly but once you start widening that interface it probably becomes more of an issue. Just something to consider.
One case, as you pointed out, is when you having multiple versions of same plugin and want to swap them on the fly.
Yeah, but that use case is the least convincing to me. We can do pretty much what you describe below (with the minor variation that it has to be an external plugin) and generally, the replacement of plugins has historically not played much of a role. For all practical intents and purposes it seems safer to subclass/switch the primitives in the image instead of replacing the plugin.
Another case, is when you want to overlook some activity by installing a proxy which will log all calls to some functions.
Yes, I could see doing that for debugging/auditing/tracing/profiling purposes. That's definitely a more interesting use case.
Third case:
Smalltalk unloadModule: 'UnwantedStuff'.
if there another hideous plugin (or VM) which using ioLoadFunctionFrom(), it would be very hard to prevent from loading it again :)
I don't get this use case. Can you elaborate?
Another one, as variant of first case (OldBlt/NewBlt), suppose you have module 'fooBar' which using functionality of some 'defaultFoo' module. But as well, you have a 'highEndFoo' module, loaded as well which having same interface as defaultFoo, but slightly different implementation:
Smalltalk lowerCPUUsage: bool bool ifTrue: [ HighEndFoo deactivate ] ifFalse: [ HighEndFoo reactivate ]
Note, that here, #deactivate wont lead to unloading a module. It stays in memory, ready to be used later, but simply rerouting things back to 'defaultFoo'.
Again, I find this particular use case not very convincing. In a situation like the above I would prefer having two classes in the image referring to the plugins explicitly and have my code use whichever it finds appropriate - because in this case you can run both the low-end as well as the high-end side-by-side and code won't stomp on each other by one deciding it wants to use the low-end and the other one deciding to use the high-end version.
I'm not saying that there's much pressing need in such use cases. But these 3 functions will give us more flexibility , security and modularity.
How so? More flexibility, yes probably. More security? Most definitely not - plugins run at a trusted level already so an evil plugin can do bad things and nothing of what you're describing will prevent that. Modularity I'd say is arguable; but given how badly Smalltalk fares with a flat global namespace I'd say that you'll be hard pressed to explain how using a single flat namespace is an improvement over a per-plugin namespace ;-)
To be fair , i'd make all plugins to export only single function, like setInterpreter(), which then tells VM what things it wants to expose. Currently, all plugins exposing functions and primitives in static manner - and have no control at run time what to expose or not. But that's another story :)
Well, actually it's intrinsically related. Part of that story is that plugins use dlopen/dlsym for the lookup from an external plugin. It's because of that that the exports have to be static. So in order to make this dynamic you would have to have a single entry exporting it.
Cheers, - Andreas
2008/11/22 Andreas Raab andreas.raab@gmx.de:
Igor Stasenko wrote:
The main reason why to use the flat namespace is to allow sharing state identified by name between modules without exact knowledge where this state comes from. This, of course implies some contracts about meaning of symbols , e.g. "bitblt" is a function pointer which takes N arguments and so on (but hey, you implying the same contract anyways, when calling ioLoadFunctionFrom("ioBitBlt", "BitBltPlugin") - just with one exception, that you have to explicitly specify the source module where it comes from).
Well, what's interesting about this is that it becomes much more of an issue if this mechanism is widely used to share function pointers. Right now, 99% of the entry points into plugins are primitives which use the Smalltalk stack and do their own argument checking. There is only a tiny number of functions that is used directly but once you start widening that interface it probably becomes more of an issue. Just something to consider.
Of course you right. What i proposed is simplest possible enhancement. A more secure design can be employed, if there such need. But there always a risk of abuse regardless how much obstacles you put - its C with direct memory access after all :) All you need to do to be in safe is to use correct function prototypes. But same applies to functions which you importing from dynamic library.
One case, as you pointed out, is when you having multiple versions of same plugin and want to swap them on the fly.
Yeah, but that use case is the least convincing to me. We can do pretty much what you describe below (with the minor variation that it has to be an external plugin) and generally, the replacement of plugins has historically not played much of a role. For all practical intents and purposes it seems safer to subclass/switch the primitives in the image instead of replacing the plugin.
Another case, is when you want to overlook some activity by installing a proxy which will log all calls to some functions.
Yes, I could see doing that for debugging/auditing/tracing/profiling purposes. That's definitely a more interesting use case.
Third case:
Smalltalk unloadModule: 'UnwantedStuff'.
if there another hideous plugin (or VM) which using ioLoadFunctionFrom(), it would be very hard to prevent from loading it again :)
I don't get this use case. Can you elaborate?
It comes from implementation of ioLoadFunctionFrom() - it there is no module loaded , it tries to load it first.
Another one, as variant of first case (OldBlt/NewBlt), suppose you have module 'fooBar' which using functionality of some 'defaultFoo' module. But as well, you have a 'highEndFoo' module, loaded as well which having same interface as defaultFoo, but slightly different implementation:
Smalltalk lowerCPUUsage: bool bool ifTrue: [ HighEndFoo deactivate ] ifFalse: [ HighEndFoo reactivate ]
Note, that here, #deactivate wont lead to unloading a module. It stays in memory, ready to be used later, but simply rerouting things back to 'defaultFoo'.
Again, I find this particular use case not very convincing. In a situation like the above I would prefer having two classes in the image referring to the plugins explicitly and have my code use whichever it finds appropriate - because in this case you can run both the low-end as well as the high-end side-by-side and code won't stomp on each other by one deciding it wants to use the low-end and the other one deciding to use the high-end version.
I'm not saying that there's much pressing need in such use cases. But these 3 functions will give us more flexibility , security and modularity.
How so? More flexibility, yes probably. More security? Most definitely not - plugins run at a trusted level already so an evil plugin can do bad things and nothing of what you're describing will prevent that. Modularity I'd say is arguable; but given how badly Smalltalk fares with a flat global namespace I'd say that you'll be hard pressed to explain how using a single flat namespace is an improvement over a per-plugin namespace ;-)
By more security, i meant a little feature, that since getting atom value is inexpensive operation, you can load value identified by atom and check for non-null value each time before calling function. Old scheme, needs to check for non-null as well, but in addition you need to be careful to clean out this pointer when you get notification that module, from where you taken this pointer is unloaded.
I'm not sure what you mean by per-plugin namespace. And how much difference in namespaces between this: bitblt = ioLoadFunctionFrom("ioBitBlt", "BitBltPlugin") and this: atom = makeAtom("BitBltPlugin.ioBitBlt"); bitBlt = getAtomValue(atom);
The difference that first binds symbol at compile/link time, while second - at run time.
As for modularity, let me illustrate what i meant. There are many primitives in VM, which look like:
primitiveFoo: x with: y
^ self cCode: 'ioFoo(x,y)'.
Obviously, when you generate this code using VMMaker, it wont compile unless you having ioFoo() function implemented somewhere. And you have a little choice where to put this implementation - in one of internal plugins or in platform-specific part of VM.
Now, lets consider, if i refactor this code to use atoms (i skipping details like declarations etc) :
primitiveFoo: x with: y
ioFoo := getAtomValue: AtomIoFoo. ioFoo notNil ifTrue: [ ^ self cCode: 'ioFoo(x,y)' ]. ^ self primitiveFail.
1. the code will compile regardless i declared a function or not. 2. i'm free in choice where to put the implementation of this function, or even build VM initially w/o this function. 3. across different platforms, one can use same VMMaker to generate sources , and it will always compile & link.
doesn't it look like more modular?
Now, take a look at a sqWin32Stubs.c - how it would look if we would use shared namespace? It would look as zero length file. Because all we have to do in platform code is just initialize pointers:
pointers = { "ioFoo" , ioFoo #ifdef NO_SOME_STUFF "ioBar" , ioBar #endif ... };
in this way, if i don't want some stuff , i can simply comment out a line in this table and let code rot in sources :)
I'm currently trying to make a HostWindowsPlugin and want to put all windowing and i/o stuff in it from VM platform sources. It would be much a cut'n'paste experience to me, if all callouts in VM would use atoms - because then i don't need to touch headers & sources and many different places to make compiler content. Because there would be no need in exporting function in C-like manner.
And lastly, remember InterpreterProxy thing? Each time you want to introduce new functionality , you have to extend the structure and increase version number. But what is it? Its just a list of pointers! Isn't it would be simpler for plugin to just lookup a function by its name - and use it if such function exists. Its pretty much same as dynamic linking, just s/dlsym/getAtomValue/ and don't let your code be constrained by compiler/linker anymore :)
To be fair , i'd make all plugins to export only single function, like setInterpreter(), which then tells VM what things it wants to expose. Currently, all plugins exposing functions and primitives in static manner - and have no control at run time what to expose or not. But that's another story :)
Well, actually it's intrinsically related. Part of that story is that plugins use dlopen/dlsym for the lookup from an external plugin. It's because of that that the exports have to be static. So in order to make this dynamic you would have to have a single entry exporting it.
Actually the idea to use it came to my mind after reading this: http://tronche.com/gui/x/icccm/sec-1.html#s-1 . They calling it inter-client communications :) As well, as hated SystemDictionary does pretty same thing :)
Cheers,
- Andreas
Igor Stasenko wrote:
By more security, i meant a little feature, that since getting atom value is inexpensive operation, you can load value identified by atom and check for non-null value each time before calling function.
Oh, I think you mean it can be safer to use (with which I would agree) not necessarily more secure.
Old scheme, needs to check for non-null as well, but in addition you need to be careful to clean out this pointer when you get notification that module, from where you taken this pointer is unloaded.
Yes. Although, you won't get around doing something about unloading either - we spent today learning about the intricacies of unloading OpenAL32.dll and it turned out to be absolutely crucial to be able to unload our own plugins. A flat shared namespace might have caused some serious issues here.
I'm not sure what you mean by per-plugin namespace. And how much difference in namespaces between this: bitblt = ioLoadFunctionFrom("ioBitBlt", "BitBltPlugin") and this: atom = makeAtom("BitBltPlugin.ioBitBlt"); bitBlt = getAtomValue(atom);
The difference that first binds symbol at compile/link time, while second - at run time.
It would be quite possible to bind this at runtime, too. But what I mean by per-plugin namespace is that the *export* isn't done into a flat namespace but rather into a structured one.
And yes, one could conceivably use a naming scheme that is equivalent in practice but unless that's automated (the current scheme is fully automated) it seems error-prone and one of these things were people are simply too lazy in practice to utilize it correctly (if that were different I would expect people to use class and selector prefixes consistently which they don't).
As for modularity, let me illustrate what i meant. There are many primitives in VM, which look like:
primitiveFoo: x with: y
^ self cCode: 'ioFoo(x,y)'.
Obviously, when you generate this code using VMMaker, it wont compile unless you having ioFoo() function implemented somewhere. And you have a little choice where to put this implementation - in one of internal plugins or in platform-specific part of VM.
Now, lets consider, if i refactor this code to use atoms (i skipping details like declarations etc) :
primitiveFoo: x with: y
ioFoo := getAtomValue: AtomIoFoo. ioFoo notNil ifTrue: [ ^ self cCode: 'ioFoo(x,y)' ]. ^ self primitiveFail.
- the code will compile regardless i declared a function or not.
- i'm free in choice where to put the implementation of this
function, or even build VM initially w/o this function. 3. across different platforms, one can use same VMMaker to generate sources , and it will always compile & link.
doesn't it look like more modular?
No. It looks utterly pointless to me. You introduce a plugin that does nothing but looking up and call an atom; what good is that plugin? If you generalize that just a little you have the FFI where you might declare ioFoo() directly and call it. Which of course could be done via atom table too, but I still fail to see how that would be more modular.
Now, take a look at a sqWin32Stubs.c - how it would look if we would use shared namespace? It would look as zero length file. Because all we have to do in platform code is just initialize pointers:
pointers = { "ioFoo" , ioFoo #ifdef NO_SOME_STUFF "ioBar" , ioBar #endif ... };
And if you stick this in sqWin32Stubs.c (or its equivalent) you end up with a non-empty stubs file. In other words, you are replacing one set of stubs with another one. Not much of an improvement.
I'm currently trying to make a HostWindowsPlugin and want to put all windowing and i/o stuff in it from VM platform sources.
Why do another one? Is there something that the current host window plugin doesn't address?
It would be much a cut'n'paste experience to me, if all callouts in VM would use atoms - because then i don't need to touch headers & sources and many different places to make compiler content. Because there would be no need in exporting function in C-like manner.
Ah. But that's a fallacy. That you don't need to "touch" these places doesn't mean you don't need to know about them. In fact, having to touch them, having the compiler complain about them is a great way to learn and know and understand all the areas you need to touch - if the VM would randomly crash on you because you have replaced one but not another function you'd be in for a hellish experience. Yes, the C compiler can be notorious but it can also be a pretty good teacher.
And lastly, remember InterpreterProxy thing? Each time you want to introduce new functionality , you have to extend the structure and increase version number. But what is it? Its just a list of pointers!
No, it's not. It's an interface (a contract) between the VM and the plugin.
Isn't it would be simpler for plugin to just lookup a function by its name - and use it if such function exists. Its pretty much same as dynamic linking, just s/dlsym/getAtomValue/ and don't let your code be constrained by compiler/linker anymore :)
I *very much* doubt that it would be simpler for each plugin to look up the function every time they use it and test for its existence every time it is used. Consider this primitive:
primitiveStringClone interpreterProxy methodArgument = 1 ifFalse:[^interpreterProxy primitiveFail]. arg := interpreterProxy stackValue: 0. (interpreterProxy isBytes: arg) ifFalse:[^interpreterProxy primitiveFail]. clone := interpreterProxy clone: arg. interpreterProxy pop: 2. interpreterProxy push: clone.
Now let's rewrite this in pseudo-code to see what it would look like without an interface:
primitiveStringClone ((interpreterProxy has: #methodArgumentCount) and:[interpreterProxy methodArgument = 1]) ifFalse:[^interpreterProxy primitiveFail]. (interpreterProxy has: #stackValue) ifFalse:[^interpreterProxy primitiveFail]. arg := interpreterProxy stackValue: 0. (interpreterProxy has: #isBytes) ifFalse:[^interpreterProxy primitiveFail]. (interpreterProxy isBytes: arg) ifFalse:[^interpreterProxy primitiveFail]. (interpreterProxy has: #clone) ifFalse:[^interpreterProxy primitiveFail]. clone := interpreterProxy clone: arg. (interpreterProxy has: #pop) ifFalse:[^interpreterProxy primitiveFail]. interpreterProxy pop: 2. (interpreterProxy has: #push) ifFalse:[^interpreterProxy primitiveFail]. interpreterProxy push: clone.
Simpler? You've got to be kidding me ;-)
Cheers, - Andreas
2008/11/22 Andreas Raab andreas.raab@gmx.de:
Igor Stasenko wrote:
By more security, i meant a little feature, that since getting atom value is inexpensive operation, you can load value identified by atom and check for non-null value each time before calling function.
Oh, I think you mean it can be safer to use (with which I would agree) not necessarily more secure.
Old scheme, needs to check for non-null as well, but in addition you need to be careful to clean out this pointer when you get notification that module, from where you taken this pointer is unloaded.
Yes. Although, you won't get around doing something about unloading either - we spent today learning about the intricacies of unloading OpenAL32.dll and it turned out to be absolutely crucial to be able to unload our own plugins. A flat shared namespace might have caused some serious issues here.
Almost anything "might have cause some serious issues", if you don't use it wisely (Class become: nil). Its not an argument not to use it.
I'm not sure what you mean by per-plugin namespace. And how much difference in namespaces between this: bitblt = ioLoadFunctionFrom("ioBitBlt", "BitBltPlugin") and this: atom = makeAtom("BitBltPlugin.ioBitBlt"); bitBlt = getAtomValue(atom);
The difference that first binds symbol at compile/link time, while second - at run time.
It would be quite possible to bind this at runtime, too. But what I mean by per-plugin namespace is that the *export* isn't done into a flat namespace but rather into a structured one.
And yes, one could conceivably use a naming scheme that is equivalent in practice but unless that's automated (the current scheme is fully automated) it seems error-prone and one of these things were people are simply too lazy in practice to utilize it correctly (if that were different I would expect people to use class and selector prefixes consistently which they don't).
It could be automated or used manually. Its open for any intrusion :)
As for modularity, let me illustrate what i meant. There are many primitives in VM, which look like:
primitiveFoo: x with: y
^ self cCode: 'ioFoo(x,y)'.
Obviously, when you generate this code using VMMaker, it wont compile unless you having ioFoo() function implemented somewhere. And you have a little choice where to put this implementation - in one of internal plugins or in platform-specific part of VM.
Now, lets consider, if i refactor this code to use atoms (i skipping details like declarations etc) :
primitiveFoo: x with: y
ioFoo := getAtomValue: AtomIoFoo. ioFoo notNil ifTrue: [ ^ self cCode: 'ioFoo(x,y)' ]. ^ self primitiveFail.
- the code will compile regardless i declared a function or not.
- i'm free in choice where to put the implementation of this
function, or even build VM initially w/o this function. 3. across different platforms, one can use same VMMaker to generate sources , and it will always compile & link.
doesn't it look like more modular?
No. It looks utterly pointless to me. You introduce a plugin that does nothing but looking up and call an atom; what good is that plugin? If you generalize that just a little you have the FFI where you might declare ioFoo() directly and call it. Which of course could be done via atom table too, but I still fail to see how that would be more modular.
Not a plugin. I mentioned VM code in interp.c. I see little need in having such constructs in plugin. In VM core, however, there is always a bit of uncertainty - VM forced to provide primitive by default (because its a standart one), but on different platforms, depending on their capabilities or your intent, you may omit putting functionality of some primitives into VM. This is where such scheme is can be quite useful - instead of stubbing function in sources, or exploring what function has to return to make primitive fail , just remove it from build.
Now, take a look at a sqWin32Stubs.c - how it would look if we would use shared namespace? It would look as zero length file. Because all we have to do in platform code is just initialize pointers:
pointers = { "ioFoo" , ioFoo #ifdef NO_SOME_STUFF "ioBar" , ioBar #endif ... };
And if you stick this in sqWin32Stubs.c (or its equivalent) you end up with a non-empty stubs file. In other words, you are replacing one set of stubs with another one. Not much of an improvement.
Its not the same thing. Currently you have to define a function - with implementation or with empty body to make compiler content. In example i showed - you can do simpler - just omit binding a function to a symbol. Suppose we having a well structured organization of VM platform sources, then it might look like:
#ifdef SUPPORT_FILES #include "file_functions.inc" /// or put all functions here w/o using #include #endif
instead of something like:
#ifdef SUPPORT_FILES #include "file_functions.inc" /// or put all functions here w/o using #include #else #include "file_functions_stubs.inc" #endif
I'm currently trying to make a HostWindowsPlugin and want to put all windowing and i/o stuff in it from VM platform sources.
Why do another one? Is there something that the current host window plugin doesn't address?
Well, there's many things. Don't want to OT here, just simple example: when image fires up, i want to show a splash screen, and then, when user clicks on it, or some time passed , hide it and show the "main" window. ALL is controlled from image, of course , e.g. one might want to show splash screen and when user clicks on it - just quit the squeak :)
It would be much a cut'n'paste experience to me, if all callouts in VM would use atoms - because then i don't need to touch headers & sources and many different places to make compiler content. Because there would be no need in exporting function in C-like manner.
Ah. But that's a fallacy. That you don't need to "touch" these places doesn't mean you don't need to know about them. In fact, having to touch them, having the compiler complain about them is a great way to learn and know and understand all the areas you need to touch - if the VM would randomly crash on you because you have replaced one but not another function you'd be in for a hellish experience. Yes, the C compiler can be notorious but it can also be a pretty good teacher.
The same self-fallacy is to be confident, that if your code compiles ok it doesn't crash in some random place :)
And lastly, remember InterpreterProxy thing? Each time you want to introduce new functionality , you have to extend the structure and increase version number. But what is it? Its just a list of pointers!
No, it's not. It's an interface (a contract) between the VM and the plugin.
Isn't it would be simpler for plugin to just lookup a function by its name - and use it if such function exists. Its pretty much same as dynamic linking, just s/dlsym/getAtomValue/ and don't let your code be constrained by compiler/linker anymore :)
I *very much* doubt that it would be simpler for each plugin to look up the function every time they use it and test for its existence every time it is used. Consider this primitive:
primitiveStringClone interpreterProxy methodArgument = 1 ifFalse:[^interpreterProxy primitiveFail]. arg := interpreterProxy stackValue: 0. (interpreterProxy isBytes: arg) ifFalse:[^interpreterProxy primitiveFail]. clone := interpreterProxy clone: arg. interpreterProxy pop: 2. interpreterProxy push: clone.
Now let's rewrite this in pseudo-code to see what it would look like without an interface:
primitiveStringClone ((interpreterProxy has: #methodArgumentCount) and:[interpreterProxy methodArgument = 1]) ifFalse:[^interpreterProxy primitiveFail]. (interpreterProxy has: #stackValue) ifFalse:[^interpreterProxy primitiveFail]. arg := interpreterProxy stackValue: 0. (interpreterProxy has: #isBytes) ifFalse:[^interpreterProxy primitiveFail]. (interpreterProxy isBytes: arg) ifFalse:[^interpreterProxy primitiveFail]. (interpreterProxy has: #clone) ifFalse:[^interpreterProxy primitiveFail]. clone := interpreterProxy clone: arg. (interpreterProxy has: #pop) ifFalse:[^interpreterProxy primitiveFail]. interpreterProxy pop: 2. (interpreterProxy has: #push) ifFalse:[^interpreterProxy primitiveFail]. interpreterProxy push: clone.
Simpler? You've got to be kidding me ;-)
This counter-example missing a point :) Take a look at Hydra code, where InterpreterProxy has left with only 3 functions, and will stay to have 3 functions forever without need in extending protocol , because rest functions is obtained dynamicaly using name lookup. A plugin simply refuse to start without having all requested VM functions be non-null. But the trick is, that VM not forced anymore to support obsolete stuff of ever-growing InterpreterProxy protocol.
Cheers,
- Andreas
On Sat, Nov 22, 2008 at 9:34 AM, Igor Stasenko siguctua@gmail.com wrote:
2008/11/22 Andreas Raab andreas.raab@gmx.de:
Igor Stasenko wrote:
By more security, i meant a little feature, that since getting atom value is inexpensive operation, you can load value identified by atom and check for non-null value each time before calling function.
Oh, I think you mean it can be safer to use (with which I would agree)
not
necessarily more secure.
Old scheme, needs to check for non-null as well, but in addition you need to be careful to clean out this pointer when you get notification that module, from where you taken this pointer is unloaded.
Yes. Although, you won't get around doing something about unloading
either -
we spent today learning about the intricacies of unloading OpenAL32.dll
and
it turned out to be absolutely crucial to be able to unload our own
plugins.
A flat shared namespace might have caused some serious issues here.
Almost anything "might have cause some serious issues", if you don't use it wisely (Class become: nil). Its not an argument not to use it.
I'm not sure what you mean by per-plugin namespace. And how much difference in namespaces between this: bitblt = ioLoadFunctionFrom("ioBitBlt", "BitBltPlugin") and this: atom = makeAtom("BitBltPlugin.ioBitBlt"); bitBlt = getAtomValue(atom);
The difference that first binds symbol at compile/link time, while second - at run time.
It would be quite possible to bind this at runtime, too. But what I mean
by
per-plugin namespace is that the *export* isn't done into a flat
namespace
but rather into a structured one.
And yes, one could conceivably use a naming scheme that is equivalent in practice but unless that's automated (the current scheme is fully
automated)
it seems error-prone and one of these things were people are simply too
lazy
in practice to utilize it correctly (if that were different I would
expect
people to use class and selector prefixes consistently which they don't).
It could be automated or used manually. Its open for any intrusion :)
As for modularity, let me illustrate what i meant. There are many primitives in VM, which look like:
primitiveFoo: x with: y
^ self cCode: 'ioFoo(x,y)'.
Obviously, when you generate this code using VMMaker, it wont compile unless you having ioFoo() function implemented somewhere. And you have a little choice where to put this implementation - in one of internal plugins or in platform-specific part of VM.
Now, lets consider, if i refactor this code to use atoms (i skipping details like declarations etc) :
primitiveFoo: x with: y
ioFoo := getAtomValue: AtomIoFoo. ioFoo notNil ifTrue: [ ^ self cCode: 'ioFoo(x,y)' ]. ^ self primitiveFail.
- the code will compile regardless i declared a function or not.
- i'm free in choice where to put the implementation of this
function, or even build VM initially w/o this function. 3. across different platforms, one can use same VMMaker to generate sources , and it will always compile & link.
doesn't it look like more modular?
No. It looks utterly pointless to me. You introduce a plugin that does nothing but looking up and call an atom; what good is that plugin? If you generalize that just a little you have the FFI where you might declare ioFoo() directly and call it. Which of course could be done via atom
table
too, but I still fail to see how that would be more modular.
Not a plugin. I mentioned VM code in interp.c. I see little need in having such constructs in plugin. In VM core, however, there is always a bit of uncertainty - VM forced to provide primitive by default (because its a standart one), but on different platforms, depending on their capabilities or your intent, you may omit putting functionality of some primitives into VM. This is where such scheme is can be quite useful - instead of stubbing function in sources, or exploring what function has to return to make primitive fail , just remove it from build.
Now, take a look at a sqWin32Stubs.c - how it would look if we would use shared namespace? It would look as zero length file. Because all we have to do in platform code is just initialize pointers:
pointers = { "ioFoo" , ioFoo #ifdef NO_SOME_STUFF "ioBar" , ioBar #endif ... };
And if you stick this in sqWin32Stubs.c (or its equivalent) you end up
with
a non-empty stubs file. In other words, you are replacing one set of
stubs
with another one. Not much of an improvement.
Its not the same thing. Currently you have to define a function - with implementation or with empty body to make compiler content. In example i showed - you can do simpler - just omit binding a function to a symbol. Suppose we having a well structured organization of VM platform sources, then it might look like:
#ifdef SUPPORT_FILES #include "file_functions.inc" /// or put all functions here w/o using #include #endif
instead of something like:
#ifdef SUPPORT_FILES #include "file_functions.inc" /// or put all functions here w/o using #include #else #include "file_functions_stubs.inc" #endif
I'm currently trying to make a HostWindowsPlugin and want to put all windowing and i/o stuff in it from VM platform sources.
Why do another one? Is there something that the current host window
plugin
doesn't address?
Well, there's many things. Don't want to OT here, just simple example: when image fires up, i want to show a splash screen, and then, when user clicks on it, or some time passed , hide it and show the "main" window. ALL is controlled from image, of course , e.g. one might want to show splash screen and when user clicks on it - just quit the squeak :)
It would be much a cut'n'paste experience to me, if all callouts in VM would use atoms - because then i don't need to touch headers & sources and many different places to make compiler content. Because there would be no need in exporting function in C-like manner.
Ah. But that's a fallacy. That you don't need to "touch" these places doesn't mean you don't need to know about them. In fact, having to touch them, having the compiler complain about them is a great way to learn and know and understand all the areas you need to touch - if the VM would randomly crash on you because you have replaced one but not another
function
you'd be in for a hellish experience. Yes, the C compiler can be
notorious
but it can also be a pretty good teacher.
The same self-fallacy is to be confident, that if your code compiles ok it doesn't crash in some random place :)
And lastly, remember InterpreterProxy thing? Each time you want to introduce new functionality , you have to extend the structure and increase version number. But what is it? Its just a list of pointers!
No, it's not. It's an interface (a contract) between the VM and the
plugin.
Isn't it would be simpler for plugin to just lookup a function by its name - and use it if such function exists. Its pretty much same as dynamic linking, just s/dlsym/getAtomValue/ and don't let your code be constrained by compiler/linker anymore :)
I *very much* doubt that it would be simpler for each plugin to look up
the
function every time they use it and test for its existence every time it
is
used. Consider this primitive:
primitiveStringClone interpreterProxy methodArgument = 1 ifFalse:[^interpreterProxy primitiveFail]. arg := interpreterProxy stackValue: 0. (interpreterProxy isBytes: arg) ifFalse:[^interpreterProxy primitiveFail]. clone := interpreterProxy clone: arg. interpreterProxy pop: 2. interpreterProxy push: clone.
Now let's rewrite this in pseudo-code to see what it would look like
without
an interface:
primitiveStringClone ((interpreterProxy has: #methodArgumentCount) and:[interpreterProxy methodArgument = 1]) ifFalse:[^interpreterProxy primitiveFail]. (interpreterProxy has: #stackValue) ifFalse:[^interpreterProxy primitiveFail]. arg := interpreterProxy stackValue: 0. (interpreterProxy has: #isBytes) ifFalse:[^interpreterProxy primitiveFail]. (interpreterProxy isBytes: arg) ifFalse:[^interpreterProxy primitiveFail]. (interpreterProxy has: #clone) ifFalse:[^interpreterProxy primitiveFail]. clone := interpreterProxy clone: arg. (interpreterProxy has: #pop) ifFalse:[^interpreterProxy primitiveFail]. interpreterProxy pop: 2. (interpreterProxy has: #push) ifFalse:[^interpreterProxy primitiveFail]. interpreterProxy push: clone.
Simpler? You've got to be kidding me ;-)
This counter-example missing a point :) Take a look at Hydra code, where InterpreterProxy has left with only 3 functions, and will stay to have 3 functions forever without need in extending protocol , because rest functions is obtained dynamicaly using name lookup. A plugin simply refuse to start without having all requested VM functions be non-null. But the trick is, that VM not forced anymore to support obsolete stuff of ever-growing InterpreterProxy protocol.
I like this very much. Not totally, just very much :)
What's good here is that the plugin can access functions directly in the VM and not go through the slow interpreterProxy. Because C supports syntax for calls through function pointers that looks just like normal functions calls wouldn't look very different to normal calls. The function pointers simply need to be decared.
But what I don't like is having interpreterProxy persist at all. Why not get rid of it except for initialization and pass it in through setInterpreter? What do you need interpreterProxy to persist for?
So Slang generates a standard prolog for all the functions used in a plugin at the beginning of the file (nice documentation; lists exactly the functions the plugin uses). setInterpreter (better called initializePlugin?) takes an argument that provides a function to lookup function pointers by name. Slang generates the code to initialize these fnction pointers. Uses of the function pointers look just like normal calls.
Then internal plugins can be linked directly against the VM. e.g.
#if EXTERNAL_PLUGIN static void (*popthenPush(sqInt,sqInt); static void (*primitiveFailed)(void); #endif ... void somePrimitive() { .... if (!ok) { primitiveFailed(); return; } popthenPush(result); } ... #if EXTERNAL_PLUGIN static sqInt setInterpreter(InterpreterProxy *interpreterProxy) { if (interpreterProxy.oopSize() != 4) return InitErrorWrongOopSize; if (!interpreterProxy.getFunction("popthenPush", &popthenPush) || !interpreterProxy.getFunction("primitiveFailed", & primitiveFailed)) return InitErrorMissingFunction; return 0; } #endif /* EXTERNAL_PLUGIN */
Cheers,
- Andreas
-- Best regards, Igor Stasenko AKA sig.
2008/11/22 Eliot Miranda eliot.miranda@gmail.com:
On Sat, Nov 22, 2008 at 9:34 AM, Igor Stasenko siguctua@gmail.com wrote:
2008/11/22 Andreas Raab andreas.raab@gmx.de:
Igor Stasenko wrote:
By more security, i meant a little feature, that since getting atom value is inexpensive operation, you can load value identified by atom and check for non-null value each time before calling function.
Oh, I think you mean it can be safer to use (with which I would agree) not necessarily more secure.
Old scheme, needs to check for non-null as well, but in addition you need to be careful to clean out this pointer when you get notification that module, from where you taken this pointer is unloaded.
Yes. Although, you won't get around doing something about unloading either - we spent today learning about the intricacies of unloading OpenAL32.dll and it turned out to be absolutely crucial to be able to unload our own plugins. A flat shared namespace might have caused some serious issues here.
Almost anything "might have cause some serious issues", if you don't use it wisely (Class become: nil). Its not an argument not to use it.
I'm not sure what you mean by per-plugin namespace. And how much difference in namespaces between this: bitblt = ioLoadFunctionFrom("ioBitBlt", "BitBltPlugin") and this: atom = makeAtom("BitBltPlugin.ioBitBlt"); bitBlt = getAtomValue(atom);
The difference that first binds symbol at compile/link time, while second - at run time.
It would be quite possible to bind this at runtime, too. But what I mean by per-plugin namespace is that the *export* isn't done into a flat namespace but rather into a structured one.
And yes, one could conceivably use a naming scheme that is equivalent in practice but unless that's automated (the current scheme is fully automated) it seems error-prone and one of these things were people are simply too lazy in practice to utilize it correctly (if that were different I would expect people to use class and selector prefixes consistently which they don't).
It could be automated or used manually. Its open for any intrusion :)
As for modularity, let me illustrate what i meant. There are many primitives in VM, which look like:
primitiveFoo: x with: y
^ self cCode: 'ioFoo(x,y)'.
Obviously, when you generate this code using VMMaker, it wont compile unless you having ioFoo() function implemented somewhere. And you have a little choice where to put this implementation - in one of internal plugins or in platform-specific part of VM.
Now, lets consider, if i refactor this code to use atoms (i skipping details like declarations etc) :
primitiveFoo: x with: y
ioFoo := getAtomValue: AtomIoFoo. ioFoo notNil ifTrue: [ ^ self cCode: 'ioFoo(x,y)' ]. ^ self primitiveFail.
- the code will compile regardless i declared a function or not.
- i'm free in choice where to put the implementation of this
function, or even build VM initially w/o this function. 3. across different platforms, one can use same VMMaker to generate sources , and it will always compile & link.
doesn't it look like more modular?
No. It looks utterly pointless to me. You introduce a plugin that does nothing but looking up and call an atom; what good is that plugin? If you generalize that just a little you have the FFI where you might declare ioFoo() directly and call it. Which of course could be done via atom table too, but I still fail to see how that would be more modular.
Not a plugin. I mentioned VM code in interp.c. I see little need in having such constructs in plugin. In VM core, however, there is always a bit of uncertainty - VM forced to provide primitive by default (because its a standart one), but on different platforms, depending on their capabilities or your intent, you may omit putting functionality of some primitives into VM. This is where such scheme is can be quite useful - instead of stubbing function in sources, or exploring what function has to return to make primitive fail , just remove it from build.
Now, take a look at a sqWin32Stubs.c - how it would look if we would use shared namespace? It would look as zero length file. Because all we have to do in platform code is just initialize pointers:
pointers = { "ioFoo" , ioFoo #ifdef NO_SOME_STUFF "ioBar" , ioBar #endif ... };
And if you stick this in sqWin32Stubs.c (or its equivalent) you end up with a non-empty stubs file. In other words, you are replacing one set of stubs with another one. Not much of an improvement.
Its not the same thing. Currently you have to define a function - with implementation or with empty body to make compiler content. In example i showed - you can do simpler - just omit binding a function to a symbol. Suppose we having a well structured organization of VM platform sources, then it might look like:
#ifdef SUPPORT_FILES #include "file_functions.inc" /// or put all functions here w/o using #include #endif
instead of something like:
#ifdef SUPPORT_FILES #include "file_functions.inc" /// or put all functions here w/o using #include #else #include "file_functions_stubs.inc" #endif
I'm currently trying to make a HostWindowsPlugin and want to put all windowing and i/o stuff in it from VM platform sources.
Why do another one? Is there something that the current host window plugin doesn't address?
Well, there's many things. Don't want to OT here, just simple example: when image fires up, i want to show a splash screen, and then, when user clicks on it, or some time passed , hide it and show the "main" window. ALL is controlled from image, of course , e.g. one might want to show splash screen and when user clicks on it - just quit the squeak :)
It would be much a cut'n'paste experience to me, if all callouts in VM would use atoms - because then i don't need to touch headers & sources and many different places to make compiler content. Because there would be no need in exporting function in C-like manner.
Ah. But that's a fallacy. That you don't need to "touch" these places doesn't mean you don't need to know about them. In fact, having to touch them, having the compiler complain about them is a great way to learn and know and understand all the areas you need to touch - if the VM would randomly crash on you because you have replaced one but not another function you'd be in for a hellish experience. Yes, the C compiler can be notorious but it can also be a pretty good teacher.
The same self-fallacy is to be confident, that if your code compiles ok it doesn't crash in some random place :)
And lastly, remember InterpreterProxy thing? Each time you want to introduce new functionality , you have to extend the structure and increase version number. But what is it? Its just a list of pointers!
No, it's not. It's an interface (a contract) between the VM and the plugin.
Isn't it would be simpler for plugin to just lookup a function by its name - and use it if such function exists. Its pretty much same as dynamic linking, just s/dlsym/getAtomValue/ and don't let your code be constrained by compiler/linker anymore :)
I *very much* doubt that it would be simpler for each plugin to look up the function every time they use it and test for its existence every time it is used. Consider this primitive:
primitiveStringClone interpreterProxy methodArgument = 1 ifFalse:[^interpreterProxy primitiveFail]. arg := interpreterProxy stackValue: 0. (interpreterProxy isBytes: arg) ifFalse:[^interpreterProxy primitiveFail]. clone := interpreterProxy clone: arg. interpreterProxy pop: 2. interpreterProxy push: clone.
Now let's rewrite this in pseudo-code to see what it would look like without an interface:
primitiveStringClone ((interpreterProxy has: #methodArgumentCount) and:[interpreterProxy methodArgument = 1]) ifFalse:[^interpreterProxy primitiveFail]. (interpreterProxy has: #stackValue) ifFalse:[^interpreterProxy primitiveFail]. arg := interpreterProxy stackValue: 0. (interpreterProxy has: #isBytes) ifFalse:[^interpreterProxy primitiveFail]. (interpreterProxy isBytes: arg) ifFalse:[^interpreterProxy primitiveFail]. (interpreterProxy has: #clone) ifFalse:[^interpreterProxy primitiveFail]. clone := interpreterProxy clone: arg. (interpreterProxy has: #pop) ifFalse:[^interpreterProxy primitiveFail]. interpreterProxy pop: 2. (interpreterProxy has: #push) ifFalse:[^interpreterProxy primitiveFail]. interpreterProxy push: clone.
Simpler? You've got to be kidding me ;-)
This counter-example missing a point :) Take a look at Hydra code, where InterpreterProxy has left with only 3 functions, and will stay to have 3 functions forever without need in extending protocol , because rest functions is obtained dynamicaly using name lookup. A plugin simply refuse to start without having all requested VM functions be non-null. But the trick is, that VM not forced anymore to support obsolete stuff of ever-growing InterpreterProxy protocol.
I like this very much. Not totally, just very much :) What's good here is that the plugin can access functions directly in the VM and not go through the slow interpreterProxy. Because C supports syntax for calls through function pointers that looks just like normal functions calls wouldn't look very different to normal calls. The function pointers simply need to be decared.
right!
But what I don't like is having interpreterProxy persist at all. Why not get rid of it except for initialization and pass it in through setInterpreter? What do you need interpreterProxy to persist for?
right!
So Slang generates a standard prolog for all the functions used in a plugin at the beginning of the file (nice documentation; lists exactly the functions the plugin uses). setInterpreter (better called initializePlugin?) takes an argument that provides a function to lookup function pointers by name. Slang generates the code to initialize these fnction pointers. Uses of the function pointers look just like normal calls.
Right!
Then internal plugins can be linked directly against the VM. e.g.
Things works exactly in that way in Hydra. :)
#if EXTERNAL_PLUGIN static void (*popthenPush(sqInt,sqInt); static void (*primitiveFailed)(void); #endif ... void somePrimitive() { .... if (!ok) { primitiveFailed(); return; } popthenPush(result); } ... #if EXTERNAL_PLUGIN static sqInt setInterpreter(InterpreterProxy *interpreterProxy) { if (interpreterProxy.oopSize() != 4) return InitErrorWrongOopSize; if (!interpreterProxy.getFunction("popthenPush", &popthenPush) || !interpreterProxy.getFunction("primitiveFailed", & primitiveFailed)) return InitErrorMissingFunction; return 0; } #endif /* EXTERNAL_PLUGIN */
Cheers,
- Andreas
-- Best regards, Igor Stasenko AKA sig.
OK, good :) So Igor, what are the three functions interpreterProxy still has? TIA
(P.S. this can be backward-compatible, right? We can still provide the old nterface for a while to allow external plugins to still load)
On Sat, Nov 22, 2008 at 11:06 AM, Igor Stasenko siguctua@gmail.com wrote:
2008/11/22 Eliot Miranda eliot.miranda@gmail.com:
On Sat, Nov 22, 2008 at 9:34 AM, Igor Stasenko siguctua@gmail.com
wrote:
2008/11/22 Andreas Raab andreas.raab@gmx.de:
Igor Stasenko wrote:
By more security, i meant a little feature, that since getting atom value is inexpensive operation, you can load value identified by atom and check for non-null value each time before calling function.
Oh, I think you mean it can be safer to use (with which I would agree) not necessarily more secure.
Old scheme, needs to check for non-null as well, but in addition you need to be careful to clean out this pointer when you get
notification
that module, from where you taken this pointer is unloaded.
Yes. Although, you won't get around doing something about unloading either - we spent today learning about the intricacies of unloading
OpenAL32.dll
and it turned out to be absolutely crucial to be able to unload our own plugins. A flat shared namespace might have caused some serious issues here.
Almost anything "might have cause some serious issues", if you don't use it wisely (Class become: nil). Its not an argument not to use it.
I'm not sure what you mean by per-plugin namespace. And how much difference in namespaces between this: bitblt = ioLoadFunctionFrom("ioBitBlt", "BitBltPlugin") and this: atom = makeAtom("BitBltPlugin.ioBitBlt"); bitBlt = getAtomValue(atom);
The difference that first binds symbol at compile/link time, while second - at run time.
It would be quite possible to bind this at runtime, too. But what I
mean
by per-plugin namespace is that the *export* isn't done into a flat namespace but rather into a structured one.
And yes, one could conceivably use a naming scheme that is equivalent
in
practice but unless that's automated (the current scheme is fully automated) it seems error-prone and one of these things were people are simply
too
lazy in practice to utilize it correctly (if that were different I would expect people to use class and selector prefixes consistently which they don't).
It could be automated or used manually. Its open for any intrusion :)
As for modularity, let me illustrate what i meant. There are many primitives in VM, which look like:
primitiveFoo: x with: y
^ self cCode: 'ioFoo(x,y)'.
Obviously, when you generate this code using VMMaker, it wont compile unless you having ioFoo() function implemented somewhere. And you have a little choice where to put this implementation - in
one
of internal plugins or in platform-specific part of VM.
Now, lets consider, if i refactor this code to use atoms (i skipping details like declarations etc) :
primitiveFoo: x with: y
ioFoo := getAtomValue: AtomIoFoo. ioFoo notNil ifTrue: [ ^ self cCode: 'ioFoo(x,y)' ]. ^ self primitiveFail.
- the code will compile regardless i declared a function or not.
- i'm free in choice where to put the implementation of this
function, or even build VM initially w/o this function. 3. across different platforms, one can use same VMMaker to generate sources , and it will always compile & link.
doesn't it look like more modular?
No. It looks utterly pointless to me. You introduce a plugin that does nothing but looking up and call an atom; what good is that plugin? If you generalize that just a little you have the FFI where you might declare ioFoo() directly and call it. Which of course could be done via atom table too, but I still fail to see how that would be more modular.
Not a plugin. I mentioned VM code in interp.c. I see little need in having such constructs in plugin. In VM core, however, there is always a bit of uncertainty - VM forced to provide primitive by default (because its a standart one), but on different platforms, depending on their capabilities or your intent, you may omit putting functionality of some primitives into VM. This is where such scheme is can be quite useful - instead of stubbing function in sources, or exploring what function has to return to make primitive fail , just remove it from build.
Now, take a look at a sqWin32Stubs.c - how it would look if we would use shared namespace? It would look as zero length file. Because all we have to do in platform code is just initialize
pointers:
pointers = { "ioFoo" , ioFoo #ifdef NO_SOME_STUFF "ioBar" , ioBar #endif ... };
And if you stick this in sqWin32Stubs.c (or its equivalent) you end up with a non-empty stubs file. In other words, you are replacing one set of stubs with another one. Not much of an improvement.
Its not the same thing. Currently you have to define a function - with implementation or with empty body to make compiler content. In example i showed - you can do simpler - just omit binding a function to a symbol. Suppose we having a well structured organization of VM platform sources, then it might look like:
#ifdef SUPPORT_FILES #include "file_functions.inc" /// or put all functions here w/o using #include #endif
instead of something like:
#ifdef SUPPORT_FILES #include "file_functions.inc" /// or put all functions here w/o using #include #else #include "file_functions_stubs.inc" #endif
I'm currently trying to make a HostWindowsPlugin and want to put all windowing and i/o stuff in it from VM platform sources.
Why do another one? Is there something that the current host window plugin doesn't address?
Well, there's many things. Don't want to OT here, just simple example: when image fires up, i want to show a splash screen, and then, when user clicks on it, or some time passed , hide it and show the "main" window. ALL is controlled from image, of course , e.g. one might want to show splash screen and when user clicks on it - just quit the squeak :)
It would be much a cut'n'paste experience to me, if all callouts in
VM
would use atoms - because then i don't need to touch headers &
sources
and many different places to make compiler content. Because there would be no need in exporting function in C-like manner.
Ah. But that's a fallacy. That you don't need to "touch" these places doesn't mean you don't need to know about them. In fact, having to
touch
them, having the compiler complain about them is a great way to learn and know and understand all the areas you need to touch - if the VM would randomly crash on you because you have replaced one but not another function you'd be in for a hellish experience. Yes, the C compiler can be notorious but it can also be a pretty good teacher.
The same self-fallacy is to be confident, that if your code compiles ok it doesn't crash in some random place :)
And lastly, remember InterpreterProxy thing? Each time you want to introduce new functionality , you have to extend the structure and increase version number. But what is it? Its just a list of pointers!
No, it's not. It's an interface (a contract) between the VM and the plugin.
Isn't it would be simpler for plugin to just lookup a function by its name - and use it if such function exists. Its pretty much same as dynamic linking, just s/dlsym/getAtomValue/ and don't let your code
be
constrained by compiler/linker anymore :)
I *very much* doubt that it would be simpler for each plugin to look
up
the function every time they use it and test for its existence every time
it
is used. Consider this primitive:
primitiveStringClone interpreterProxy methodArgument = 1 ifFalse:[^interpreterProxy primitiveFail]. arg := interpreterProxy stackValue: 0. (interpreterProxy isBytes: arg) ifFalse:[^interpreterProxy primitiveFail]. clone := interpreterProxy clone: arg. interpreterProxy pop: 2. interpreterProxy push: clone.
Now let's rewrite this in pseudo-code to see what it would look like without an interface:
primitiveStringClone ((interpreterProxy has: #methodArgumentCount) and:[interpreterProxy methodArgument = 1]) ifFalse:[^interpreterProxy primitiveFail]. (interpreterProxy has: #stackValue) ifFalse:[^interpreterProxy primitiveFail]. arg := interpreterProxy stackValue: 0. (interpreterProxy has: #isBytes) ifFalse:[^interpreterProxy primitiveFail]. (interpreterProxy isBytes: arg) ifFalse:[^interpreterProxy primitiveFail]. (interpreterProxy has: #clone) ifFalse:[^interpreterProxy primitiveFail]. clone := interpreterProxy clone: arg. (interpreterProxy has: #pop) ifFalse:[^interpreterProxy primitiveFail]. interpreterProxy pop: 2. (interpreterProxy has: #push) ifFalse:[^interpreterProxy primitiveFail]. interpreterProxy push: clone.
Simpler? You've got to be kidding me ;-)
This counter-example missing a point :) Take a look at Hydra code, where InterpreterProxy has left with only 3 functions, and will stay to have 3 functions forever without need in extending protocol , because rest functions is obtained dynamicaly using name lookup. A plugin simply refuse to start without having all requested VM functions be non-null. But the trick is, that VM not forced anymore to support obsolete stuff of ever-growing InterpreterProxy protocol.
I like this very much. Not totally, just very much :) What's good here is that the plugin can access functions directly in the
VM
and not go through the slow interpreterProxy. Because C supports syntax
for
calls through function pointers that looks just like normal functions
calls
wouldn't look very different to normal calls. The function pointers
simply
need to be decared.
right!
But what I don't like is having interpreterProxy persist at all. Why not
get
rid of it except for initialization and pass it in through
setInterpreter?
What do you need interpreterProxy to persist for?
right!
So Slang generates a standard prolog for all the functions used in a
plugin
at the beginning of the file (nice documentation; lists exactly the functions the plugin uses). setInterpreter (better called initializePlugin?) takes an argument that provides a function to lookup function pointers by name. Slang generates the code to initialize these fnction pointers. Uses of the function pointers look just like normal calls.
Right!
Then internal plugins can be linked directly against the VM. e.g.
Things works exactly in that way in Hydra. :)
#if EXTERNAL_PLUGIN static void (*popthenPush(sqInt,sqInt); static void (*primitiveFailed)(void); #endif ... void somePrimitive() { .... if (!ok) { primitiveFailed(); return; } popthenPush(result); } ... #if EXTERNAL_PLUGIN static sqInt setInterpreter(InterpreterProxy *interpreterProxy) { if (interpreterProxy.oopSize() != 4) return InitErrorWrongOopSize; if (!interpreterProxy.getFunction("popthenPush", &popthenPush) || !interpreterProxy.getFunction("primitiveFailed", &
primitiveFailed))
return InitErrorMissingFunction; return 0;
} #endif /* EXTERNAL_PLUGIN */
Cheers,
- Andreas
-- Best regards, Igor Stasenko AKA sig.
-- Best regards, Igor Stasenko AKA sig.
2008/11/22 Eliot Miranda eliot.miranda@gmail.com:
OK, good :) So Igor, what are the three functions interpreterProxy still has?
Here the part of sqVirtualMachine.h
typedef struct ObjVirtualMachine { sqInt (*minorVersion)(void); sqInt (*majorVersion)(void);
/* IMPORTANT!!! * The rest of functions can be obtained by plugin by calling a getVMFunctionPointerBySelector function. * The need in defining additional functions in this struct is gone forever */ void * (*getVMFunctionPointerBySelector)(char * selector);
} ObjVirtualMachine;
TIA
(P.S. this can be backward-compatible, right? We can still provide the old nterface for a while to allow external plugins to still load)
Yes, first it calls setInterpreter() with new struct. Old plugin checks version and refuses to work. Then VM calls setInterpreter() with old InterpreterProxy, to make plugin happy.
On Sat, Nov 22, 2008 at 11:06 AM, Igor Stasenko siguctua@gmail.com wrote:
Hi Igor, one question Andreas raised that I hadn't thought of was how with the new scheme you deal with multiple interpreters? With the old interpreterProxy scheme I guess the interpreterProxy could redirect a function to the relevant interpreter (although this would require a mutex to make it thread safe). How do you do it in Hydra with the new scheme? Do you load the plugin multiple times, once per VM, or do you do some redrecting behind the scenes?
On Sat, Nov 22, 2008 at 11:11 AM, Igor Stasenko siguctua@gmail.com wrote:
2008/11/22 Eliot Miranda eliot.miranda@gmail.com:
OK, good :) So Igor, what are the three functions interpreterProxy still has?
Here the part of sqVirtualMachine.h
typedef struct ObjVirtualMachine { sqInt (*minorVersion)(void); sqInt (*majorVersion)(void);
/* IMPORTANT!!!
The rest of functions can be obtained by plugin by calling a
getVMFunctionPointerBySelector function.
The need in defining additional functions in this struct is gone
forever */ void * (*getVMFunctionPointerBySelector)(char * selector);
} ObjVirtualMachine;
TIA
(P.S. this can be backward-compatible, right? We can still provide the
old
nterface for a while to allow external plugins to still load)
Yes, first it calls setInterpreter() with new struct. Old plugin checks version and refuses to work. Then VM calls setInterpreter() with old InterpreterProxy, to make plugin happy.
On Sat, Nov 22, 2008 at 11:06 AM, Igor Stasenko siguctua@gmail.com
wrote:
-- Best regards, Igor Stasenko AKA sig.
2008/11/26 Eliot Miranda eliot.miranda@gmail.com:
Hi Igor, one question Andreas raised that I hadn't thought of was how with the new scheme you deal with multiple interpreters? With the old interpreterProxy scheme I guess the interpreterProxy could redirect a function to the relevant interpreter (although this would require a mutex to make it thread safe). How do you do it in Hydra with the new scheme? Do you load the plugin multiple times, once per VM, or do you do some redrecting behind the scenes?
To maintain backward compatibility in Hydra VM we need to support two kinds of plugins: old and new ones.
We have a limitation , that old plugins could work only with single interpreter - main one. Any attempt to use primitives of old plugin in non-main interpreter will lead to primitive failure. Since we can't predict what plugin does, or what internal state its using for work - its really dangerous to use its primitives in a concurrent world. So, we saying big NO to use old plugins for non-main interpreter. In this way, we guarantee that Hydra VM works with old plugins identically as Squeak VM, but only for the main interpreter.
Of course, things are different for new plugins: they are aware that their primitives could run concurrently in multiple threads.
In concurrent world we have to make difference between global state and thread-local(per-interpreter) state. To access global state, there is nothing new - we should use synchronization objects (mutex/semaphore). But most plugins which i converted to Hydra don't using a mutable global state, or initialize it once (at plugin loading stage) and using it in read-only manner. So, the major thing is how to deal with per-interpreter state of plugins. My solution is simple: i added helper functions into VM which allows users to 'attach' a state buffer to all interpreter instances.
By calling #attachStateBuffer: numBytes initializeFn: fn1 finalizeFn: fn2 , plugin telling VM, that for each interpreter instance it needs a numBytes long memory buffer to hold its thread-local state. In result , it receives a unique state id, which it should use later to access an attached state buffer of particular interpreter instance.
So each time it needs to access own thread-local state, it simply calls #getAttachedStateBuffer: aStateHandle of: interpreter
And , of course plugin can provide two pointers of functions which initializing the state buffer or finalize it. These functions is called each time you creating new instance of interpreter or destroying it.
In Hydra-s VMMaker, i made modifications which seamlessly (for plugin writer) deals with per-interpreter state. As well as in old generation scheme - all you need is to declare some ivars in plugin's class. But code generator , instead of placing them at module scope level , wrapping all such variables with struct.
I think it not requires too much explanation, for you Eliot, but for other readers:
Let me show you the difference between old and new code generation on an example of B2DPlugin.
The old generator simply puts all variables into C file:
/*** Variables ***/ static int* aetBuffer; static char bbPluginName[256] = "BitBltPlugin"; static void * copyBitsFn; static sqInt dispatchReturnValue; static sqInt dispatchedValue; static int doProfileStats = 0; static sqInt engine; static sqInt engineStopped; static sqInt formArray; static sqInt geProfileTime; static int* getBuffer;
#ifdef SQUEAK_BUILTIN_PLUGIN extern #endif struct VirtualMachine* interpreterProxy; static void * loadBBFn; static const char *moduleName = #ifdef SQUEAK_BUILTIN_PLUGIN "B2DPlugin 11 November 2008 (i)" #else "B2DPlugin 11 November 2008 (e)" #endif ; static int* objBuffer; static sqInt objUsed; static unsigned int* spanBuffer; static int* workBuffer;
The new one is a bit more clever: all global variables still going to C file, but thread-local going to special struct, declared in <plugin-name>_imports.h :
/* Per-image plugin state */ typedef struct PluginState { int* workBuffer; int* objBuffer; int* getBuffer; int* aetBuffer; unsigned int* spanBuffer; sqInt engine; sqInt formArray; sqInt engineStopped; sqInt geProfileTime; sqInt dispatchedValue; sqInt dispatchReturnValue; sqInt objUsed; #ifdef PLUGIN_STATE_EXTRAS PLUGIN_STATE_EXTRAS #endif } PluginState;
And it defines two helper macros:
#define DECLARE_PLUGIN_STATE() struct PluginState * pstate = vmFunction(getAttachedStateBufferof)(pluginStateId,currentInterpreter()) #define plugin_state(name) pstate->name
Now lets take a look at method code, generated by VMMaker:
/* old */ static sqInt addEdgeToGET(sqInt edge) { if (!(allocateGETEntry(1))) { return 0; } getBuffer[workBuffer[GWGETUsed]] = edge; workBuffer[GWGETUsed] = ((workBuffer[GWGETUsed]) + 1); }
/* new */ static sqInt addEdgeToGET _iargs(sqInt edge) { DECLARE_PLUGIN_STATE(); if (!(allocateGETEntry _iparams(1))) { return 0; } plugin_state(getBuffer)[plugin_state(workBuffer)[GWGETUsed]] = edge; plugin_state(workBuffer)[GWGETUsed] = ((plugin_state(workBuffer)[GWGETUsed]) + 1); }
here an _iargs() macro expands to #define _iargs(args...) (PInterpreter intr, args) and currentInterpreter() macro expands to 'intr'
So, in expanded form code could look like:
static sqInt addEdgeToGET (PInterpreter intr, sqInt edge) { PluginState * pstate = getAttachedStateBufferof(pluginStateId, intr);
pstate->foo = blabla }
Few words about vmFunction(name)
#ifdef SQUEAK_BUILTIN_PLUGIN
#include "interp_prototypes.h" #define vmFunction(name) name
#else
#define vmFunction(name) vmFunctions.name
so, as you see , this macro expands to 'name' in internal plugins and YES, it calls vm functions directly w/o using interpreterProxy.
As for external plugin its expands to an a vmFunctions.name , which is a struct in a form:
typedef struct VMFunctions { char * sel1; sqInt (*attachStateBufferinitializeFnfinalizeFn) (sqInt numberOfBytes, AttachedStateFn stateInitializeFn, AttachedStateFn stateFinalizeFn); char * sel2; sqInt (*booleanValueOf) _iargs(sqInt obj); char * sel3; sqInt (*byteSizeOf) (sqInt oop); char * sel4; sqInt (*classBitmap) _iarg(); char * sel5; sqInt (*classPoint) _iarg(); char * sel6; sqInt (*failed) _iarg(); ... void * _dummy; void * _dummyfn; }
In C file we got a variable vmFunctions
struct VMFunctions vmFunctions = { "attachStateBuffer:initializeFn:finalizeFn:", NULL, "booleanValueOf:", NULL, "byteSizeOf:", NULL, "classBitmap", NULL, "classPoint", NULL, "failed", NULL, "fetchClassOf:", NULL, "fetchInteger:ofObject:", NULL, "fetchPointer:ofObject:", NULL, "firstIndexableField:", NULL, ..... NULL, NULL };
And in, setInterpreter() - a first function which called by VM, this structure is filled with appropriate pointers:
EXPORT(sqInt) setInterpreter (InterpreterProxy * anInterpreter) { sqInt i; char * ptr; char** table; sqInt ok;
ok = anInterpreter->majorVersion() == OBJVM_PROXY_MAJOR; if (ok == 0) { return 0; } ok = anInterpreter->minorVersion() >= OBJVM_PROXY_MINOR; if (ok == 0) { return 0; } i = 0; table = (char**) &vmFunctions; while ((table[i]) != null) { ptr = (char*)anInterpreter->getVMFunctionPointerBySelector(table[i]); if (ptr == null) { dprintf(("Plugin unable to find vm function: %s\n", table[i])); return 0; } table[i + 1] = ptr; i += 2; } ; ok = INIT_PLUGIN_STATE();
return ok; }
as you can see, plugin fails to initialize (returns 0) if any of functions its using is null.
There is more things under the hood concerning where i got function prototypes , and how VM generates a list of public functions.. but i think its enough for this post :)
.. continuing In the light of current topic about shared namespace. Things could be more uniform:
let suppose VM having following functions:
int makeAtom(char * atomName) -- which returns an atom id, and interns an unique name (in same way as symbols interned in squeak) , and sets it default value to null (if it wasn't internet before). Any subsequent calls to this function with same argument will return same id.
Two accessor functions: void * getAtomValue(int atomId); void * setAtomValue(int atomId, void * newValue); /* returns old value */
And, surely we need a special thread-local storage
makeThreadLocalAtom(char * atomName, initFn, destroyFn)
this function behaves similar to makeAtom() except that getAtomValue/setAtomValue will use thread-local storage to access atom value. InitFn/destroyFn is a functions in a form:
void Fn(Interpreter * intr, int atomId);
Not sure about init/destroy functions. Maybe they not needed, but instead a VM should notify all plugins about creating new interpreter or destroying existing one, so plugins should care for themselves about initializing/finalizing/allocating/deallocating their state. This just a choice between, where to put a handling logic - into VM, or into each plugin.
Now, how things would look like if we apply unification thoughout VM:
1. VM could use atoms to define own public function pointers:
setAtom(makeAtom("KERNEL.fetchClassOf:") , & fetchClassOf);
then external plugin don't needs interpreterProxy. All it have to do is:
fetchClassOfFnPtr = getAtomValue(makeAtom("KERNEL.fetchClassOf:"));
and to make a call it simply could use: fetchClassOfFnPtr (foo, bar);
2. Plugin state (Hydra specific):
int stateId = makeThreadLocalAtom("BitBltPlugin.TLS", initFn, destroyFn);
and to access it: struct PluginState* pstate = (struct PluginState* ) getAtomValue(stateId);
and, of course we can use similar macros for it DECLARE_STATE() / pstate(name)
so, things remain pretty same for code generation - only different macros.
3. Primitives.
Plugins at loading stage should:
- register atom "modulaName" and set its value to non-nil. e.g. setAtomValue(makeAtom("BitBltPlugin"), 1); or.. we could put a version there.. not really relevant.
Trough this atom, we letting know VM that plugin is loaded , so it shouldn't try to find/load module again. It tries to load a module only if atom value is null.
Then, as you may suggest, primitive pointers registered by plugin into a namespace by defining an atom names in a form: "<moduleName>.<primitiveName>", and setting their values.
If VM encounters an unknown primitive (primitiveIndex is 0, but method has a named primitive) it does following: first, checks if there is non-null atom value with given module name
moduleLoaded = getAtomValue(makeAtom(moduleName)); if (!moduleLoaded) ioLoadModule(...blabla)
then instead of storing a direct function pointer for given primitive, into primitiveCache, it puts an atom id for "<moduleName>.<primitiveName>" name.
so, that primitiveCache[primitiveIndex] = atomId and to get function pointer it should read it with getAtomValue().
In this way, a plugin may override primitive pointer values at run time (by setting atom value to null or to different function), while for VM its a simple task: before call, fetch atom value by id, call the primitive or report failure if value is null.
Its really helpful for debugging/monitoring purposes. As i described earlier, one plugin may override a primitive pointers of another plugin, for collecting a different statistical information or for logging activity etc etc. Of course such usage introducing some risks, but as our folks says: the one who not risking, not drinking a champagne :)
Igor Stasenko wrote:
Yes. Although, you won't get around doing something about unloading either - we spent today learning about the intricacies of unloading OpenAL32.dll and it turned out to be absolutely crucial to be able to unload our own plugins. A flat shared namespace might have caused some serious issues here.
Almost anything "might have cause some serious issues", if you don't use it wisely (Class become: nil). Its not an argument not to use it.
Certainly. And it wasn't an argument not to use it, merely a practical point about something that comes up with a flat shared namespace (cleaning up that namespace in the face of unloading modules and possibly overwriting entries that are now owned by other modules).
doesn't it look like more modular?
No. It looks utterly pointless to me. You introduce a plugin that does nothing but looking up and call an atom; what good is that plugin? If you generalize that just a little you have the FFI where you might declare ioFoo() directly and call it. Which of course could be done via atom table too, but I still fail to see how that would be more modular.
Not a plugin. I mentioned VM code in interp.c. I see little need in having such constructs in plugin. In VM core, however, there is always a bit of uncertainty - VM forced to provide primitive by default (because its a standart one), but on different platforms, depending on their capabilities or your intent, you may omit putting functionality of some primitives into VM.
Well, that's why we came up with plugins to begin with ;-) So that different platforms and different builds can have different sets of functionality.
Its not the same thing. Currently you have to define a function - with implementation or with empty body to make compiler content. In example i showed - you can do simpler - just omit binding a function to a symbol.
Sure. But it's still a completely contrived example. The stubs are only there because once upon a time, in an age long past there weren't VM plugins and you couldn't leave out entire plugins from the generated VM so the stubs were a necessity. I need to remove these stubs - they are completely pointless in this day and age. Thanks for reminding me.
Take a look at Hydra code, where InterpreterProxy has left with only 3 functions, and will stay to have 3 functions forever without need in extending protocol , because rest functions is obtained dynamicaly using name lookup. A plugin simply refuse to start without having all requested VM functions be non-null. But the trick is, that VM not forced anymore to support obsolete stuff of ever-growing InterpreterProxy protocol.
I see - I had understood you wrongly. I thought you were explicitly trying to back out of the role of InterpreterProxy as the contract (which is not fulfilled if the VM doesn't export all the required functions in Hydra) and rather have the plugin code deal with the necessary lookups manually.
The way Hydra does the proxy lookup is just fine as far as I am concerned. There is no difference from the client's point of view - for the plugin writer the proxy remains the contract and the ability to selectively allow and deny certain entry points can in fact be helpful since it allows a more flexible set of definitions of what is part of a particular proxy interface.
Cheers, - Andreas
2008/11/22 Andreas Raab andreas.raab@gmx.de:
Igor Stasenko wrote:
Yes. Although, you won't get around doing something about unloading either - we spent today learning about the intricacies of unloading OpenAL32.dll and it turned out to be absolutely crucial to be able to unload our own plugins. A flat shared namespace might have caused some serious issues here.
Almost anything "might have cause some serious issues", if you don't use it wisely (Class become: nil). Its not an argument not to use it.
Certainly. And it wasn't an argument not to use it, merely a practical point about something that comes up with a flat shared namespace (cleaning up that namespace in the face of unloading modules and possibly overwriting entries that are now owned by other modules).
I agree, it is not flawless, as well as current implementation. I just looking for a ways to relax interdependency between different modules or between VM and modules. A shared namespace fits good for some cases listed, and not so good for others. No wonder, there can't be single tool to make everything :) If you see better alternatives (to shared namespace), i'll listen with interest.
doesn't it look like more modular?
No. It looks utterly pointless to me. You introduce a plugin that does nothing but looking up and call an atom; what good is that plugin? If you generalize that just a little you have the FFI where you might declare ioFoo() directly and call it. Which of course could be done via atom table too, but I still fail to see how that would be more modular.
Not a plugin. I mentioned VM code in interp.c. I see little need in having such constructs in plugin. In VM core, however, there is always a bit of uncertainty - VM forced to provide primitive by default (because its a standart one), but on different platforms, depending on their capabilities or your intent, you may omit putting functionality of some primitives into VM.
Well, that's why we came up with plugins to begin with ;-) So that different platforms and different builds can have different sets of functionality.
Yeah, but still there are places which is quite rigid and causing headache when you have to deal with them. For instance, try building VM without including SecurityPlugin:
//F/projects/pharo-freetype/src32/winbuild/../platforms/win32/vm/sqWin32Intel.c:1128: undefined reference to `ioInitSecurity' ./obj/vm/sqWin32Prefs.o: In function `SetAllowFileAccess': //F/projects/pharo-freetype/src32/winbuild/../platforms/win32/vm/sqWin32Prefs.c:96: undefined reference to `ioHasFileAccess' ./obj/vm/sqWin32Prefs.o: In function `SetAllowImageWrite': //F/projects/pharo-freetype/src32/winbuild/../platforms/win32/vm/sqWin32Prefs.c:101: undefined reference to `ioCanWriteImage' ./obj/vm/sqWin32Prefs.o: In function `SetAllowSocketAccess': //F/projects/pharo-freetype/src32/winbuild/../platforms/win32/vm/sqWin32Prefs.c:106: undefined reference to `ioHasSocketAccess' ./obj/vm/sqWin32Prefs.o: In function `HandlePrefsMenu': //F/projects/pharo-freetype/src32/winbuild/../platforms/win32/vm/sqWin32Prefs.c:383: undefined reference to `ioHasFileAccess' //F/projects/pharo-freetype/src32/winbuild/../platforms/win32/vm/sqWin32Prefs.c:383: undefined reference to `_ioSetFileAccess' //F/projects/pharo-freetype/src32/winbuild/../platforms/win32/vm/sqWin32Prefs.c:387: undefined reference to `ioCanWriteImage' //F/projects/pharo-freetype/src32/winbuild/../platforms/win32/vm/sqWin32Prefs.c:387: undefined reference to `_ioSetImageWrite' //F/projects/pharo-freetype/src32/winbuild/../platforms/win32/vm/sqWin32Prefs.c:391: undefined reference to `ioHasSocketAccess' //F/projects/pharo-freetype/src32/winbuild/../platforms/win32/vm/sqWin32Prefs.c:391: undefined reference to `_ioSetSocketAccess' -------
Without SocketPlugin:
./obj/vm/sqWin32Prefs.o: In function `HandlePrefsMenu': //F/projects/pharo-freetype/src32/winbuild/../platforms/win32/vm/sqWin32Prefs.c:400: undefined reference to `win32DebugPrintSocketState' -------
Without FilePlugin (this list quite long, so i just leave a one error per single .c file):
//F/projects/pharo-freetype/src32/winbuild/./src/vm/gnu-interp.c:3242: undefined reference to `sqImageFileSeek'
//F/projects/pharo-freetype/src32/winbuild/../platforms/win32/vm/sqWin32Args.c:26: undefined reference to `sqImageFileOpen'
//F/projects/pharo-freetype/src32/winbuild/../platforms/win32/vm/sqWin32Intel.c:1116: undefined reference to `sqImageFileOpen'
//F/projects/pharo-freetype/src32/winbuild/../platforms/win32/vm/sqWin32PluginSupport.c:252: undefined reference to `fileRecordSize'
./obj/vm/AsynchFilePlugin.lib(sqWin32AsyncFilePrims.o): In function `asyncFileValid': //F/projects/pharo-freetype/src32/winbuild/obj/AsynchFilePlugin/../../../platforms/Win32/plugins/AsynchFilePlugin/sqWin32AsyncFilePrims.c:167: undefined reference to `thisSession'
./obj/vm/DropPlugin.lib(sqWin32Drop.o): In function `dropRequestFileHandle': //F/projects/pharo-freetype/src32/winbuild/obj/DropPlugin/../../../platforms/Win32/plugins/DropPlugin/sqWin32Drop.c:745: undefined reference to `fileRecordSize' ---------------
I cannot accept any reasoning, why VM would be considered defunc without any of the above plugins, and therefore they ought to be included in build. In the end it defeats the purpose of plugins: be able to plug-in or plug-out at will.
Its not the same thing. Currently you have to define a function - with implementation or with empty body to make compiler content. In example i showed - you can do simpler - just omit binding a function to a symbol.
Sure. But it's still a completely contrived example. The stubs are only there because once upon a time, in an age long past there weren't VM plugins and you couldn't leave out entire plugins from the generated VM so the stubs were a necessity. I need to remove these stubs - they are completely pointless in this day and age. Thanks for reminding me.
Yeah, less cruft is good :)
Take a look at Hydra code, where InterpreterProxy has left with only 3 functions, and will stay to have 3 functions forever without need in extending protocol , because rest functions is obtained dynamicaly using name lookup. A plugin simply refuse to start without having all requested VM functions be non-null. But the trick is, that VM not forced anymore to support obsolete stuff of ever-growing InterpreterProxy protocol.
I see - I had understood you wrongly. I thought you were explicitly trying to back out of the role of InterpreterProxy as the contract (which is not fulfilled if the VM doesn't export all the required functions in Hydra) and rather have the plugin code deal with the necessary lookups manually.
The way Hydra does the proxy lookup is just fine as far as I am concerned. There is no difference from the client's point of view - for the plugin writer the proxy remains the contract and the ability to selectively allow and deny certain entry points can in fact be helpful since it allows a more flexible set of definitions of what is part of a particular proxy interface.
Cheers,
- Andreas
On Sat, Nov 22, 2008 at 09:14:57AM +0200, Igor Stasenko wrote:
I'm currently trying to make a HostWindowsPlugin and want to put all windowing and i/o stuff in it from VM platform sources. It would be much a cut'n'paste experience to me, if all callouts in VM would use atoms - because then i don't need to touch headers & sources and many different places to make compiler content. Because there would be no need in exporting function in C-like manner.
I don't see how changing the interface to plugins helps with this problem. Any way you look at it, you have to reorganize the current one-window implementation in the platform support code as well as in the plugin. Changing the way you call plugins or obtain function addresses does not make that problem any easier. It seems to me that it adds a layer of complexity without helping to solve the problem at hand.
Actually the idea to use it came to my mind after reading this: http://tronche.com/gui/x/icccm/sec-1.html#s-1 . They calling it inter-client communications :)
There are a lot of good ideas you can borrow from X11, but complexity of implemention is not one of them.
Dave
p.s. To quote from Wikipedia: "The ICCCM is notorious for being ambiguous and difficult to correctly implement".
On 21-Nov-2008, at 11:50 AM, Igor Stasenko wrote:
I was pretty unhappy with some places in VM, where it loads a function from plugin by calling ioLoadFunction... Also, even more painful to see when one plugin wants to call another plugin function.
I know very little about the VM, but I agree what you describe is ugly and inelegant, even at a very generic level.
Now, plugins first, should declare the atoms they would want to use or provide. This can be done once at plugin initialization stage, for instance:
static sqInt bitBlitAtom = makeAtom('ioBitBlt');
Now, after acquiring atom, plugin can register a service fn under this atom:
registerService(bitBlitAtom, (void*) bitBlt); and upon unloading unregisterService(bitBlitAtom);
now, any other plugin can do:
bitBltFn = getService(bitBlitAtom);
Everything looked good until I got here.
How is another plugin supposed to access this 'bitBlitAtom' handle that is effectively always in the first plugin's private storage and literally (as you've declared it above) also in the other plugin module's private symbol namespace? (i.e. if it's compiled as a "static" variable in the first plugin module then there's no way the second plugin can use the same symbol to refer to the same storage)
Perhaps you meant to add a getAtom() function to your API? Then the second plugin would first do:
static sqInt theAtom;
theAtom = getAtom("ioBitBlt");
And then it could do the call above as:
if (theAtom != 0) bitBltFn = getService(theAtom);
if (bitBltFn) { bitBltFn( a, b,c blabla); }
squeak-dev@lists.squeakfoundation.org