Hi all.
Now by this stage you're probably all thoroughly sick of namespaces.
I've put together a more complex namespaces architecture. It's now available on the PackageUniverses for 3.10. Note that this release is still very buggy and for curious developers only.
To install: Go to Tools/NamespaceBrowser and install that.
Then browse to PasteUpMorph>>isSafeToServe, modify and save the source there. (I noticed this bug just now).
Then evaluate: "NamespaceExamples openPackageManager."
This converts existing code in the image and eventually pops up a PackageManager with four packages: Kernel, Collections, Morphic and Legacy (which contains the SystemDictionary). Right-click on these and click "Browse" and play around some. Then close Squeak, chuck the image away and wait for the next announcement :-). Currently there's a NamespaceBrowser, NamespaceWorkspace, and a package management tool where all the action starts from.
Some documentation for my approach is at http://gulik.pbwiki.com/Namespaces.
Some advantages of this approach:
- It doesn't need the SystemDictionary, but will work happily alongside it. It makes a couple of changes to the compiler etc, but these should not affect the workings of the rest of Squeak. Classes in namespaces are not stored in the SystemDictionary but are stored in packages (which again aren't stored in the SystemDictionary either).
- You can load two (or more) versions of the same package into the same image at the same time and make objects from either. Loading a package does not modify any other classes currently in the image.
- Packages depends on other packages by UUID, and if you modify a package it gets a new UUID. This means that your code will never suffer from dependency problems and will always run in exactly the same environment you wrote and tested it in. Bit-rot shouldn't happen unless you start changing the VM.
- You don't need to use haltOnce: any more. The system can run the code in a read-only package (not yet supported...) and you can do your work on a working copy of that package.
- Naming collisions are impossible (or at least they should be... not yet tested).
- It doesn't need to be integrated into the official image because it can be loaded as a Monticello package. I plan to work out a way to make .mcz or .st files that use namespaces automatically load namespace support into the image.
- It's probably compatible with Göran's approach :-). You can have two namespace implementations in your image at the same time... if you really want to.
Gulik.
Hi Michael,
I think your approach sounds great and I will definitely try it out.
I can help integrate this with Monticello.
I can see this potentially help in moving the kernel forward, since new kernel module implementations can co-exist with the old ones in the transition. I like the idea that when no one uses the old package it can simply disappear. (bye bye FileDirectory!)
For simple kernel class extensions and overrides, it is probably simplest to copy the whole class into the package. Lets call this the "copy and modify" approach.
However this could get difficult, since a package like Magma adds serialization methods to cover a lot of classes in the system, and these serialization methods do need access to all instVars. To support this there would have to be some notion of "Package A which is trusted to make extensions to Package B".... but what if I load Magma with the intention of extending all users of String, but actually users of My.String will miss out on those extensions. Also if you look at magma, you will find some classes which have method contributions from *magma-server, and contributions from *magma-client. There is a dovetailing of packages to make a whole. Perhaps this could be solved by having a specialized form of a class with is not a complete copy, but a wrapper around the original, with the extension methods. I call this the "link-to and extend" idea. Perhaps the additional methods could be defined in the form of a deltastream.
So it looks like we need a new paradigm for class extensions. If we could use deltastreams then we can have the idea of class extensions as a patch queue.
I also thought that if there is a nominated "master" repository for each namespace, then the master repository could be updated with EVERY users adaptation.
So if I were the owner of Kernel.Foo and other people include Kernel.Foo in their namespaces, whenever they "copy and modify" Kernel.Foo within their namespace, my Kernel.Foo repository gets a copy of the changed version! That way I can make an effort to integrate any additions. If users "link to and extend" my Kernel.Foo, then my repository would get a copy of the delta-stream that defines the extension.
I think that some social tools like this could help avoid the fragmentation that Göran hopes to avoid by only having one non-hierarchical namespace
Keith
On 22/09/2007, Keith Hodges keith_hodges@yahoo.co.uk wrote:
Hi Michael,
I think your approach sounds great and I will definitely try it out.
I can help integrate this with Monticello.
I can see this potentially help in moving the kernel forward, since new kernel module implementations can co-exist with the old ones in the transition. I like the idea that when no one uses the old package it can simply disappear. (bye bye FileDirectory!)
For simple kernel class extensions and overrides, it is probably simplest to copy the whole class into the package. Lets call this the "copy and modify" approach.
However this could get difficult, since a package like Magma adds serialization methods to cover a lot of classes in the system, and these serialization methods do need access to all instVars. To support this there would have to be some notion of "Package A which is trusted to make extensions to Package B".... but what if I load Magma with the intention of extending all users of String, but actually users of My.String will miss out on those extensions. Also if you look at magma, you will find some classes which have method contributions from *magma-server, and contributions from *magma-client. There is a dovetailing of packages to make a whole. Perhaps this could be solved by having a specialized form of a class with is not a complete copy, but a wrapper around the original, with the extension methods. I call this the "link-to and extend" idea. Perhaps the additional methods could be defined in the form of a deltastream.
You pointed to interesting problem. Lets consider, how should be Magma loaded into image with namespaces enabled: - any extensions to existing packages (like Kernel) should be done with 'copy and modify' approach. 'Copy and modify' can be done in two ways: - we can create a new version of base package and add new behavoirs there. Then Magma imports these modified packages, which in same turn importing old non-modified packages. Like following: Magma -> Kernel(MagmaMod) -> Kernel
Another way, we can simply copy and modify behaviors to Magma package itself and any non-modified behaviors are still visible via imports. So, magma will contain own version of, say Collection class, and don't need to change imports list.
And now most interesting thing: serializing objects. First, to make use of Magma, you need to add it to imports list in your project package. And thus, you could see all changes to base packages done by Magma.
Second, I don't think that we need to support serialization of every object in image! Personally, i think its more useful and safe to enable serialization only of those objects which used by Magma and which you making serializeable by importing Magma package into your own. In fact, since only your package knowns how to work with magma (by importing it), only this package should be granted to use serialization. Any attempt so serialize objects, which not belong to package(s) Magma imports branch should be considered as erroneous (and in fact this will lead to errors due to DNU's, because of use non-modified Kernel behaviors which installed by Magma).
Maybe this is wrong or impossible? Time will say. But i think things should be done in that way. I can see how this approach helping to prevent from appearing any objects, unrelated to your project, in Magma DB and thus could indicate design flaws.
I'm very interesting in what Michael will say. I can presume, that he strongly against enabling image-wide serialization because its obviously a security issue. And in Secure Squeak such things should be impossible.
So it looks like we need a new paradigm for class extensions. If we could use deltastreams then we can have the idea of class extensions as a patch queue.
I also thought that if there is a nominated "master" repository for each namespace, then the master repository could be updated with EVERY users adaptation.
So if I were the owner of Kernel.Foo and other people include Kernel.Foo in their namespaces, whenever they "copy and modify" Kernel.Foo within their namespace, my Kernel.Foo repository gets a copy of the changed version! That way I can make an effort to integrate any additions. If users "link to and extend" my Kernel.Foo, then my repository would get a copy of the delta-stream that defines the extension.
I think that some social tools like this could help avoid the fragmentation that Göran hopes to avoid by only having one non-hierarchical namespace
Keith
On 9/23/07, Igor Stasenko siguctua@gmail.com wrote:
On 22/09/2007, Keith Hodges keith_hodges@yahoo.co.uk wrote:
Hi Michael,
I think your approach sounds great and I will definitely try it out.
I can help integrate this with Monticello.
I tried and gave up - Monticello seems designed with a global SystemDictionary in mind and it feels like a kludge to try to make it compatible with namespaces.
Matthew Fullmer gave me a great idea on IRC yesterday; I could manually put together .mcz files. I've already got code that files out a package in chunk format, so with a bit of extra work it's possible to make a Monticello file out of it.
<snip - Keith pointing out that my approach doesn't do methods overrides>
You pointed to interesting problem.
It's interesting that you find the biggest weakness of my approach with such speed. I really don't know how to solve this, so I'm ignoring this problem until some inspiration occurs.
The main issue is that one package should not be able to directly modify another package, for security reasons, and because you lose the guarantee that packages are deployed into the same environment they were coded in.
I've thought about having some fancy technique similar to ClassBoxes, where an object has one behaviour when appearing to one client, and another behaviour when appearing to another client, but that approach is likely to open a big can of Pandora's boxes.
You could use wrapper objects. In fact, wrappers will be a common and important pattern that you would use when writing secure code.
Lets consider, how should be Magma loaded into image with namespaces
enabled:
- any extensions to existing packages (like Kernel) should be done
with 'copy and modify' approach. 'Copy and modify' can be done in two ways:
- we can create a new version of base package and add new behavoirs
there. Then Magma imports these modified packages, which in same turn importing old non-modified packages. Like following: Magma -> Kernel(MagmaMod) -> Kernel
This is a possibility, but it has a big weakness. Often you want to use instances that are passed to your package from somewhere else, and these would not be instances of your overridden class.
I also thought that if there is a nominated "master" repository for each namespace, then the master repository could be updated with EVERY users adaptation.
This would likely happen but would be done manually by developers.
So if I were the owner of Kernel.Foo and other people include Kernel.Foo
in their namespaces, whenever they "copy and modify" Kernel.Foo within their namespace, my Kernel.Foo repository gets a copy of the changed version! That way I can make an effort to integrate any additions. If users "link to and extend" my Kernel.Foo, then my repository would get a copy of the delta-stream that defines the extension.
I'm not quite this far ahead yet. This could certainly be implemented by tools if you want.
Gulik.
"Michael van der Gulik" mikevdg@gmail.com writes:
- Packages depends on other packages by UUID, and if you modify a package it
gets a new UUID. This means that your code will never suffer from dependency problems and will always run in exactly the same environment you wrote and tested it in. Bit-rot shouldn't happen unless you start changing the VM.
All dependencies work this way?
If so, then how do you handle situations where two packages depend on different versions of some third packages? Let's make it concrete and say packages A and B both depend on package Morphic. A creates a morph, hands it to B, and then B tries to add the morph to a window. How does this work? Does B's Morphic really have to cope with a morph created by A's Morphic? That would seem to mean Morphic has to be written very defensively, thus defeating the point of the exercise.
My gut is that loading multiple live versions should not be the default. By default, packages are not supposed to hum along in their own private worlds, completely immune to anything else that is happening in the image. Instead, packages are supposed to interact. They should interact with the user, with the image, and with each other.
Yes, they should interact carefully, and they should avoid making too many assumptions about each other. Still, interact they should.
Now, as a non-default option, it's a different matter. It would be cool to be able to create a sub-image that has its own versions of packages in it....
Lex
On 23 Sep 2007 14:32:40 +0200, Lex Spoon lex@lexspoon.org wrote:
"Michael van der Gulik" mikevdg@gmail.com writes:
- Packages depends on other packages by UUID, and if you modify a
package it
gets a new UUID. This means that your code will never suffer from
dependency
problems and will always run in exactly the same environment you wrote
and
tested it in. Bit-rot shouldn't happen unless you start changing the VM.
All dependencies work this way?
If so, then how do you handle situations where two packages depend on different versions of some third packages? Let's make it concrete and say packages A and B both depend on package Morphic. A creates a morph, hands it to B, and then B tries to add the morph to a window. How does this work? Does B's Morphic really have to cope with a morph created by A's Morphic? That would seem to mean Morphic has to be written very defensively, thus defeating the point of the exercise.
Yes, that would break, and this is something I hadn't thought of. Thanks for bringing it up. I guess the programmer would need to be aware of this and design code accordingly.
This makes my claim about code running in exactly the same environment false. Instead, the code just runs in a very similar environment.
My gut is that loading multiple live versions should not be the
default. By default, packages are not supposed to hum along in their own private worlds, completely immune to anything else that is happening in the image. Instead, packages are supposed to interact. They should interact with the user, with the image, and with each other.
Yes, they should interact carefully, and they should avoid making too many assumptions about each other. Still, interact they should.
The interaction is the problem -- if a package and its dependencies stays the same but its environment changes, the package may become buggy. It depends on how the package interacts with and depends on its environment. In practise I'm hoping that it won't be much of an issue. There should be a core set of stable library packages that won't change much that packages can rely on.
On another note, I've also just looked into classes in the VM's special objects array. These classes cannot be modified without causing side-effects in the rest of the VM, and you can't have multiple versions of them. I might have to add the caveat that these classes can't be modified. (These classes are: "Smalltalk specialObjectsArray select: [ :each | each isKindOf: Class ].")
Gulik.
On 23 Sep 2007 14:32:40 +0200, Lex Spoon lex@lexspoon.org wrote:
"Michael van der Gulik" mikevdg@gmail.com writes:
- Packages depends on other packages by UUID, and if you modify a package it
gets a new UUID. This means that your code will never suffer from dependency problems and will always run in exactly the same environment you wrote and tested it in. Bit-rot shouldn't happen unless you start changing the VM.
All dependencies work this way?
If so, then how do you handle situations where two packages depend on different versions of some third packages? Let's make it concrete and say packages A and B both depend on package Morphic. A creates a morph, hands it to B, and then B tries to add the morph to a window. How does this work? Does B's Morphic really have to cope with a morph created by A's Morphic? That would seem to mean Morphic has to be written very defensively, thus defeating the point of the exercise.
This is a decision which must be handled by developer. If i , lets say, creating a package and adding imports to it, then dev tools should warn me, that i have multiple versions of same package(s) imported by mine. There are two choices for developer: - update imports of those packages, so they will use same version of some base package. This is a good choice, btw, forcing developers to sync versions of different packages and update code to work fine with latest updates. - second choice is to keep things intact, then it's a developer's responsibility to write code in his package in such way, that there will no conflicts occur, like using instances of classes, created by package A, but thinking that these instances have behavior denoted in package B.
My gut is that loading multiple live versions should not be the default. By default, packages are not supposed to hum along in their own private worlds, completely immune to anything else that is happening in the image. Instead, packages are supposed to interact. They should interact with the user, with the image, and with each other.
Yes, they should interact carefully, and they should avoid making too many assumptions about each other. Still, interact they should.
Now, as a non-default option, it's a different matter. It would be cool to be able to create a sub-image that has its own versions of packages in it....
Lex
squeak-dev@lists.squeakfoundation.org
-
Igor Stasenko -
Keith Hodges -
Lex Spoon -
Michael van der Gulik