Hi Dan and all!

Dan Ingalls Dan@SqueakLand.org wrote:

Heh heh...

About six hours ago, I said I'd write a quick summary of my thoughts on the subject, and I'm just getting started. It's not that I'm writing more, but I'm trying to get it right which, as you know, takes longer.

No rush, sorry for putting on the pressure by my post. I just thought it was a good move - both with the momentum and also to show what we mean with Teams more concretely. As in - they are For Real. :)

For the record it should be said that the idea of two Teams like this originates from Dan.

Meanwhile, I'd like to proceed simply to ask for people interested in putting modules into Squeak.

I just created a new list called "modules" for this Team:

modules-subscribe@discuss.squeakfoundation.org

As part of any positive response, I'd like to hear

I am in. :)

What would you like to see in such a system?

Personally - simplicity and... simplicity.

Are you mainly interested as a designer, implementor, or user?

Partly designer/implementor, but no hardcore. Of course SM and its future place/direction/area and also my little Namespace thingy (almost scared of mentioning it nowadays :)) are my personal interests in all this.

And, unless I haven't made that absolutely clear by now, I don't have any special "standing", I don't expect my interests or proposals/solutions to be treated differently than anyone elses.

Anything else you'd like to say.

Hmmm, well, one important thing, given the "risks" with this work (we all remember the past), as with MinimalMorphic one smart idea is to run the Team "in parallell" to the regular stream as long as it can be done. Meaning that if there is a crash and burn, we aren't sitting with a half baked mess in the update stream :). But that is probably obvious.

Thanks

• Dan

PS: For now, please don't worry about overlap with current packages work. The two are closely related, and they will come together in the end, but they can be worked on independently (hopefully with cross fertilization) until then. I'm hoping to see many of the package folks among those interested.

Yes, fully agree. And I am in and I assume Lex and Alexandre are too etc.

PPS: And, lest people freak out about earthshaking changes, let me say that one of my top priorities is that, however it works, such a system should be unnoticeable to casual users of Squeak.

Yes, indeed.

regards, Göran

On Wed, 23 Feb 2005 12:36:16 -0500, Steven Swerling sswerling@yahoo.com wrote:

I'd like to see it take a Monticello package and spit out a module that loads as fast as an image segment, and permits a "postscript" ala SAR/changesets.

Heh. Some attention to fast loading might be worthwhile, yes. I had to load RB and SRP into the (admittedly sluggish) wxSqueak image, and you wouldn't say that this was a 3.2GHz machine doing the job...

On Feb 22, 2005, at 5:53 PM, Dan Ingalls wrote:

As part of any positive response, I'd like to hear

What would you like to see in such a system?

From the deployment side, I would like to see robust dependency mechanisms - for example, in Monticello there is a notion of prerequisites (ignore the fact they are changing and the Monticello isn't a package manager) that can exist on various repositories. The dependency mechanism that G-ran has talked about for the next version of SM would ostensibly allow getting the modules to be loaded from various places, not only a "central" server. Some sort of fallback mechanism for package repositories would be very useful.

Are you mainly interested as a designer, implementor, or user?

Whatever is needed :)

Anything else you'd like to say.

Thanks

• Dan

On Feb 22, 2005, at 7:53 PM, Dan Ingalls wrote:

Meanwhile, I'd like to proceed simply to ask for people interested in putting modules into Squeak.

Hi Dan,

Chalk me up as interested in modularizing Squeak, as designer, implementor and user.

One of the motivations for OmniBrowser was the observation that any modularization scheme would need tool support, and the current browser is an obstacle to that because it's difficult to modify. Indeed, the early versions of OB included package browsers based on PackageInfo packages, but I took those out on feedback from the community.

I agree with Florin that versioning is a key part of organizing Smalltalk code. One of my biggest frustrations with Monticello is that it's not been of any use to the Guides and Harvesters. That is, it's great for applications built on Squeak, but not so good for Squeak its self. I'm now working on a version of Monticello with a redesigned versioning engine, which I hope will be more useful in managing the kernel and core libraries of Squeak. More on this in another post.

I also implemented an experimental packaging system a few months ago, complete with an OB-based set of browsers. It's not quite finished, and I'm not sure it's the way to go, but it's interesting nonetheless.

I quite like Alan Lovejoy's characterization of packages as units of separately deployable code, and I hope that doesn't get overlooked in the all the discussion of versioning, categories, namespaces, dependencies and so on. All those things are necessary to some degree, since they are ways of dealing with the additional complexity that a packaged image brings with it, but they're not end goals in themselves. Separately-deployable packages is what we're after.

Some more specific things I'd like to see.

- Decoupling the organization of the image (packages, environments or whatever) from the compiler's strategy for resolving names. This will make things like compiling in a sandbox, building small images, atomically loading packages and so on much easier.

- Good tool support for packages. Browsers, debuggers, changesorters, testrunners, etc should all be package-aware.

- Good tool support by packages. With a little care, we can make core execution machinery *much* easier to manipulate, and thus make it easier to write good tools. ClassBuilder is a good example: it is very difficult to deal with any other way than through the class definition doIts that appear in the browser.

- A fast-loading and compact format for distributing code.

Thanks for taking this on, Dan. I think it's really important.

Colin

On Wednesday 23 February 2005 10:25 pm, Colin Putney wrote:

I quite like Alan Lovejoy's characterization of packages as units of separately deployable code, and I hope that doesn't get overlooked in the all the discussion of versioning, categories, namespaces, dependencies and so on. All those things are necessary to some degree, since they are ways of dealing with the additional complexity that a packaged image brings with it, but they're not end goals in themselves. Separately-deployable packages is what we're after.

Yes, and as Robert Martin points out, that means that if two packages depend on each other, then they may as well be considered parts of the same package, because you can't version them separately.

So much of the refactoring work in complicated systems amounts to removing these circular dependencies so you can develop packages at different rates.

Hi people!

"Doug Way" dway@mailcan.com wrote:

Just to throw in a concrete example, Goran & I were casually discussing (on a partitioning thread here a month ago) whether Collections should be partitioned as a separate PI package from Kernel. On one hand, you might have a separate group of maintainers taking care of Collections, versus the Kernel, which makes some sense. (At least this was the initial idea with TFNR.) But Collections is not at all separately deployable from Kernel, the two are very dependent on each other. I think the separate deployability aspect is more important, so I'd favor keeping them in the same package.

This is sort of a special case, since the Kernel is not exactly a "package" which can be loaded, it is the base image. (Although it could still be defined as a package or module.)

And thus we should also remember the idea of TFNR - the idea of making sure interested and people willing to take responsibility of a certain part should be able to do that. In this case there are probably several people interested in maintaining Collections - but not the whole kernel!

So even though I generally agree with the "deployable" attribute, for the parts that we typically can't break out yet - staking Collections out as a separate "part" (as in part-itioning) from kernel doesn't seem to be harmful. It will still be maintained using the update stream.

Sure we can lump it all together in a huge "kernel" part - and assign 20 maintainers to it - but will that give the effect we want? The "effect" being responsible motivated maintainers tending carfully for these classes. I am not sure.

Also, I'm mixing the simple near-term partitioning project with the longer-term modules project here. (Though it's not *too* long-term as Dan says. :) ) But I think the principles are the same in this case. Our partitioning should still be roughly based on deployable units.

"Rougly" yes, but... if this leads to a single "part" (the whole Basic image) then there is no point, is there? :)

• Doug

regards, Göran

Cees de Groot cg@cdegroot.com wrote:

On Fri, 25 Feb 2005 10:39:25 +0100, goran.krampe@bluefish.se wrote:

...

"Rougly" yes, but... if this leads to a single "part" (the whole Basic image) then there is no point, is there? :)

Well, yes there is. Because maybe if you're going to deploy them all at once, it's still good to partition it.

Eh, you just argued on my side - so I am guessing you misunderstood me. So I agree with all you wrote, to the letter. :)

regards, Göran

"Doug Way" dway@mailcan.com wrote:

Just to throw in a concrete example, Goran & I were casually discussing (on a partitioning thread here a month ago) whether Collections should be partitioned as a separate PI package from Kernel. On one hand, you might have a separate group of maintainers taking care of Collections, versus the Kernel, which makes some sense. (At least this was the initial idea with TFNR.) But Collections is not at all separately deployable from Kernel, the two are very dependent on each other. I think the separate deployability aspect is more important, so I'd favor keeping them in the same package.

I snipped the bit where you acknowledged that you're mixing two ideas - partitioning for development and partitioning for deployment. It's good that we make that distinction, particularly at the beginning since we now have lots of stuff that isn't separately deployable, but should be. We don't have to wait for separate deployability to divvy up maintainership of the image.

However, let's not get too used to doing that. The "separately deployable" yardstick is valuable if we stick to it ruthlessly when we make partitioning decisions. Your Kernel vs. Collections example is a great case in point. Conceptually they're really quite different, and I'd think the "feel" of maintaining them would be different as well. On the one hand, you've got really low-level stuff - the machinery of executing code in the image, interfacing with the VM, etc. On the other hand, you've got a much more abstract library, almost a distillation of mathematical concepts. They really *ought* to be separate packages, and maintained by separate groups, if only because they'll attract different kinds of people to work on them.

If we ruthlessly apply the separately-deployable rule, we may find that we'll end up with a Kernel package that includes minimal implementatons of a few core collections, and a higher-level package that embellishes the kernel collections and provides higher-level collections like OrderedCollection, SkipList and so on. There's no need to preserve the class categories as they stand today...

Of course, that kind of partitioning may prove difficult, but it will probably pay off in the end.

Colin

By the way daniel vainsencher did that with his analysis tool: goruping together packages with circular dependencies.

Stef

On 24 févr. 05, at 17:35, Ned Konz wrote:

On Wednesday 23 February 2005 10:25 pm, Colin Putney wrote:

...

I quite like Alan Lovejoy's characterization of packages as units of separately deployable code, and I hope that doesn't get overlooked in the all the discussion of versioning, categories, namespaces, dependencies and so on. All those things are necessary to some degree, since they are ways of dealing with the additional complexity that a packaged image brings with it, but they're not end goals in themselves. Separately-deployable packages is what we're after.

Yes, and as Robert Martin points out, that means that if two packages depend on each other, then they may as well be considered parts of the same package, because you can't version them separately.

So much of the refactoring work in complicated systems amounts to removing these circular dependencies so you can develop packages at different rates.

-- Ned Konz http://bike-nomad.com/squeak/

A minor correction, lest confusion creep in later...

Spoon's Naiads

At the moment, Spoon's module system is called "Naiad", but there's nothing called a "naiad" in the implementation, so pluralizing the term doesn't make sense. Rather, "Naiad" is an acronym that refers to one of the principle concepts of the system ("Name And Identity Are Distinct").

But you just might have made a typo for all I know. :)

Anyway, I would refer to it in this discussion as "Naiad (Spoon's module system)".

thanks!

-C

-- Craig Latta improvisational musical informaticist craig@netjam.org www.netjam.org [|] Proceed for Truth!

Module Folks -

I have now set up a page on the Swiki for collecting our first context. It is...

http://minnow.cc.gatech.edu/squeak/5608

Please... Clean up the format if you're handy with Swikis (I was in a rush) Fill out the content for any summary page about which you feel knowledgeable

Thanks - Dan

hi dan

since your last visit we have been looking around and this is interesting to see that over the last 4 years a lot of work have been done around a similar idea but not quite the same than our classboxes. Even if I do not believe that classboxes are the solution as is now. I think that they are interesting.

If you want I can send you some OOPSLA and ECOOP papers

For example, virtual classes proposes to introduce kind of classes extensions scope by having inheritance between modules (a hack because they could have a clean import relationship) and they have the problem that they duplicate classes from the refection point of view they have two classes. Nested Inheritance (OOPSLA 04) proposes somehow the same. The idea is that from within a module class name are lookup following some import chain (of course they have types and everything is much more complex then) but the basic principle is to late bound classname lookup so that local redefinitions takes precedence over the ones in other scopes.

mixedjuice (ECOOP 2002) they support classes extensions in Java but everything is static you cannot have different clients seen different behavior as in classboxes. So I have to recompile everything and everybody will see everything.

Again I'm not that this soemthing we want since it can make everything more complex (code navigation....)

Stef

[Well, it got a bit long, but I figured I should try to articulate my own thoughts about Modules. I think it's nice at the beginning of the project to have a few "gestalts" laid out (and thanks especially for the others so far), so that there are some anchors for discussion and so that everyone can start to build up their own vision of what's powerful and what's possible.]

Modules in Squeak I may be wrong, but I feel it is possible to solve several problems at once with a good design for modules. I also feel that it is possible to keep the modularization nearly invisible to the casual user.

What problems should a module system solve? It should support the construction of a large system out of many small ones in such a way that the parts and their relationships can be easily managed.

Partitioning One motivation for this partitioning is to reduce size and with it complexity. For example by putting many Squeak subsystems out on SqueakMap, the kernel system can be kept small and relatively easy for a novice user to comprehend.

I think it is important to keep in mind the difference between logical and physical partitioning here. For instance, we might not be concerned with size, but only with apparent complexity, so as not to overwhelm novice users. In that case, notice that the SqueakMap partitioning could be merged with the browser in such a way that a chock-full demo image could look to the user like a no-frills basic image just by changing a filter setting. But neither the logical nor the physical partitioning is easy without a supporting architecture, nor possible without a basic paradigm.

Export Once it is possible to remove and add parts of a system from and to itself, it is natural to consider moving these parts between different systems (ie, images). Two important examples of this in our current world are the the ability to import a package on SqueakMap into a new release of the system, and the ability to export a project from one system and import it into another. These examples are also important in juxtaposition -- they use completely different mechanisms to deal with boundaries, references across boundaries, and format of the stored modules. Down the road there is reason to hope that these two patterns of use would use the same mechanism.

ImageSegments, Extensions, Install and Uninstall ImageSegments, as many of you know, are a slick piece of squeak technology that allows a well-bounded network of objects to be *very* quickly exported from or imported to, a Squeak image. Well-bounded, here, means not too many in-pointers (ideally only one), and not too many out-pointers. Several years ago, we showed that the entire VM construction category, then about 500k of code, could be imported in a tenth of a second using ImageSegments. However the fact that the system is not already modular has hindered taking real advantage of this technology.

A key issue in partitioning the system is that of "extensions". By these we typically mean, and I specifically mean here, changes to shared classes that are (or should be) local to a package. A typical example would be a String method 'asURL' that makes sure it is a well-formed URL. Such a method would have no place in a system that lacked network support, so it should be a part of the NetworkSupport package rather than part of the base system. This example is fairly compelling, but even seasoned Squeakers are hard-put to say whether some certain utility methods should be included in the base classes or not.

Install and Uninstall are the processes required to install a package into, or remove it from, a host system. Typically, installation registers a number of "entry points" or names in various global tables. Our fileIn process already does much of this through its interning of Symbols and handling of global and pool variables. Uninstalling involves reversing these changes and others more problematical, such as the extensions to classes outside the package.

A Couple of Desiderata Much of my recent thinking about modules has been about how to make install and uninstall trivial. I believe it is possible (and desirable) to make an architecture in which different modules can make conflicting extensions to system classes, and in which the *only* thing required to install a module is to read it in as an imageSegment, and the only thing required to uninstall it is to nil the (only) pointer to it. A related desideratum is that, having imported a squeak project, however complex, the *only* thing required is to remove all references to it, after which it will be reclaimed by GC, and no trace of it will remain in the image. When we build a window out of a rectangle, a border width and a color, we just store three pointers; why should it be any harder to combine three modules?

Isn't this just Packages? So far, this represents my thinking of about three years ago (except I was stumped about extensions at that time), and it really reads more like "Future Packages". So the first thing I want to say is that I am in no way critical of all the great ongoing work on packages, both in package design and the effort to split up the Squeak image. Even if we come up with a better architecture for encapsulating modules in memory (and I hope we do), all of the work done so far will be relevant and useful. Moreover, I believe it is the case that the better things are partitioned, the easier it is to move from one module architecture to another.

But Wait, There's More A big shift in my thinking since that time has come from learning about E (see http://www.ERights.org). E is an architecture for secure distributed computing than can be implemented in a number of languages. Its real strengths can only be realized in pointer-safe systems, though, which makes Squeak (potentially) a very attractive host system for this architecture. The essence of E's security is modularity done right. As Mark Miller says, "security is just extreme modularity".

E in less than a Nutshell [There is a fine paper called "E in a Nutshell" by Mark Stiegler http://www.skyhunter.com/marcs/ewalnut.html ] The E work is well thought out and profound. It surrounds a module that they refer to as a vat, which can only be accessed by capabilities. I am not deeply familiar with their implementation, but I think the best way to think of capabilities in Squeak is as a set of message selectors. If a module has its own name space for selectors, and if it is impossible to forge a pointer, then if you have a module pointer and a set of selectors, then the *only* access you can possibly have to that module is to send exactly those messages.

The profundity of the E work extends to distributed computing on multiple hosts with encryption for inter-host communication, and a process model called "promise pipelining" which is like an efficient lazy publish/subscribe. They deal with race conditions, unreliable links and the whole nine yards. Moreover I believe that their model of computation also covers most of the problems of leveraging multiple processors.

A Point of High Leverage I have no delusions that we can pull off an E system in Squeak in the next six months. However my interpretation of what a module is, namely a rigorous boundary in the runtime architecture, is exactly the point at which an architecture can be made to support the requirements of E (or not). The benefits of carrying out this particular piece of work in a manner that is consistent with the requirements of E could be tremendous. For instance, imagine a 64-bit Squeak with Croquet and an airtight security model. That is why E is required reading for this project.

- Dan

On 25 févr. 05, at 19:50, Dan Ingalls wrote:

[Well, it got a bit long, but I figured I should try to articulate my own thoughts about Modules. I think it's nice at the beginning of the project to have a few "gestalts" laid out (and thanks especially for the others so far), so that there are some anchors for discussion and so that everyone can start to build up their own vision of what's powerful and what's possible.]

Modules in Squeak I may be wrong, but I feel it is possible to solve several problems at once with a good design for modules. I also feel that it is possible to keep the modularization nearly invisible to the casual user.

yes i think that this requires to change tools but having import at the level of modules and not been able to access names outside explicitly can really give the impression that you code in a box = first class category and you import stuff from other boxes. So box with imageSegments could be fun to have.

What problems should a module system solve? It should support the construction of a large system out of many small ones in such a way that the parts and their relationships can be easily managed.

Partitioning One motivation for this partitioning is to reduce size and with it complexity. For example by putting many Squeak subsystems out on SqueakMap, the kernel system can be kept small and relatively easy for a novice user to comprehend.

I think it is important to keep in mind the difference between logical and physical partitioning here. For instance, we might not be concerned with size, but only with apparent complexity, so as not to overwhelm novice users. In that case, notice that the SqueakMap partitioning could be merged with the browser in such a way that a chock-full demo image could look to the user like a no-frills basic image just by changing a filter setting. But neither the logical nor the physical partitioning is easy without a supporting architecture, nor possible without a basic paradigm.

Export Once it is possible to remove and add parts of a system from and to itself, it is natural to consider moving these parts between different systems (ie, images). Two important examples of this in our current world are the the ability to import a package on SqueakMap into a new release of the system, and the ability to export a project from one system and import it into another. These examples are also important in juxtaposition -- they use completely different mechanisms to deal with boundaries, references across boundaries, and format of the stored modules. Down the road there is reason to hope that these two patterns of use would use the same mechanism.

ImageSegments, Extensions, Install and Uninstall ImageSegments, as many of you know, are a slick piece of squeak technology that allows a well-bounded network of objects to be *very* quickly exported from or imported to, a Squeak image. Well-bounded, here, means not too many in-pointers (ideally only one), and not too many out-pointers. Several years ago, we showed that the entire VM construction category, then about 500k of code, could be imported in a tenth of a second using ImageSegments. However the fact that the system is not already modular has hindered taking real advantage of this technology.

A key issue in partitioning the system is that of "extensions". By these we typically mean, and I specifically mean here, changes to shared classes that are (or should be) local to a package. A typical example would be a String method 'asURL' that makes sure it is a well-formed URL. Such a method would have no place in a system that lacked network support, so it should be a part of the NetworkSupport package rather than part of the base system. This example is fairly compelling, but even seasoned Squeakers are hard-put to say whether some certain utility methods should be included in the base classes or not.

;) Dan I think that at the occasion this would be nice not only to have method extension but state extension.

I recently reread a good pattern from kent beck about asStuff methods. I said that you do not want to have an explosion of the interface and that you should use asStuff when the method returns an object polymorphic to the receiver. Example OrderedCollection>>asSet But you should better use URL readFrom: aString instead of String asURL. And this is was like a ahah effect for me.

Install and Uninstall are the processes required to install a package into, or remove it from, a host system. Typically, installation registers a number of "entry points" or names in various global tables. Our fileIn process already does much of this through its interning of Symbols and handling of global and pool variables. Uninstalling involves reversing these changes and others more problematical, such as the extensions to classes outside the package.

A Couple of Desiderata Much of my recent thinking about modules has been about how to make install and uninstall trivial. I believe it is possible (and desirable) to make an architecture in which different modules can make conflicting extensions to system classes, and in which the *only* thing required to install a module is to read it in as an imageSegment, and the only thing required to uninstall it is to nil the (only) pointer to it. A related desideratum is that, having imported a squeak project, however complex, the *only* thing required is to remove all references to it, after which it will be reclaimed by GC, and no trace of it will remain in the image. When we build a window out of a rectangle, a border width and a color, we just store three pointers; why should it be any harder to combine three modules?

Isn't this just Packages? So far, this represents my thinking of about three years ago (except I was stumped about extensions at that time), and it really reads more like "Future Packages". So the first thing I want to say is that I am in no way critical of all the great ongoing work on packages, both in package design and the effort to split up the Squeak image. Even if we come up with a better architecture for encapsulating modules in memory (and I hope we do), all of the work done so far will be relevant and useful. Moreover, I believe it is the case that the better things are partitioned, the easier it is to move from one module architecture to another.

But Wait, There's More A big shift in my thinking since that time has come from learning about E (see http://www.ERights.org). E is an architecture for secure distributed computing than can be implemented in a number of languages. Its real strengths can only be realized in pointer-safe systems, though, which makes Squeak (potentially) a very attractive host system for this architecture. The essence of E's security is modularity done right. As Mark Miller says, "security is just extreme modularity".

E in less than a Nutshell [There is a fine paper called "E in a Nutshell" by Mark Stiegler http://www.skyhunter.com/marcs/ewalnut.html ] The E work is well thought out and profound. It surrounds a module that they refer to as a vat, which can only be accessed by capabilities. I am not deeply familiar with their implementation, but I think the best way to think of capabilities in Squeak is as a set of message selectors. If a module has its own name space for selectors, and if it is impossible to forge a pointer, then if you have a module pointer and a set of selectors, then the *only* access you can possibly have to that module is to send exactly those messages.

The profundity of the E work extends to distributed computing on multiple hosts with encryption for inter-host communication, and a process model called "promise pipelining" which is like an efficient lazy publish/subscribe. They deal with race conditions, unreliable links and the whole nine yards. Moreover I believe that their model of computation also covers most of the problems of leveraging multiple processors.

I have the impression dan that you should invent a term for these new modules because module is connoted This is why we invented classbox. So islands or safe could be a good candidate.

A Point of High Leverage I have no delusions that we can pull off an E system in Squeak in the next six months. However my interpretation of what a module is, namely a rigorous boundary in the runtime architecture, is exactly the point at which an architecture can be made to support the requirements of E (or not). The benefits of carrying out this particular piece of work in a manner that is consistent with the requirements of E could be tremendous. For instance, imagine a 64-bit Squeak with Croquet and an airtight security model. That is why E is required reading for this project.

• Dan

Dan Ingalls wrote: <lots of good stuff snipped>

Dan, one thing that I find missing in this list is configuration management: code gets changed. So there is the issue of how to coordinate changes that different people make. One way to handle that is to partition the system, in order to distribute "ownership" and reduce conflicts. Also, I want to be sure that when I load a module, I get the same thing I got last time, independent of what is in the image before loading. And I want to be able to keep the instances while loading a different version of a module, so loading a new version is different than uninstall old, install new.

And one thought about the complexity/filtering issue: Even though the tools may be able to filter the view of the system, the code is nonetheless there and may affect the behavior of the system. So filtering may lead to "what you see is not what you get", so to say.

Just my 2c, Thanks Wolfgang

Dan Ingalls wrote:

[Well, it got a bit long, but I figured I should try to articulate my own thoughts about Modules. I think it's nice at the beginning of the project to have a few "gestalts" laid out (and thanks especially for the others so far), so that there are some anchors for discussion and so that everyone can start to build up their own vision of what's powerful and what's possible.]

Modules in Squeak I may be wrong, but I feel it is possible to solve several problems at once with a good design for modules. I also feel that it is possible to keep the modularization nearly invisible to the casual user.

What problems should a module system solve? It should support the construction of a large system out of many small ones in such a way that the parts and their relationships can be easily managed.

Partitioning One motivation for this partitioning is to reduce size and with it complexity. For example by putting many Squeak subsystems out on SqueakMap, the kernel system can be kept small and relatively easy for a novice user to comprehend.

I think it is important to keep in mind the difference between logical and physical partitioning here. For instance, we might not be concerned with size, but only with apparent complexity, so as not to overwhelm novice users. In that case, notice that the SqueakMap partitioning could be merged with the browser in such a way that a chock-full demo image could look to the user like a no-frills basic image just by changing a filter setting. But neither the logical nor the physical partitioning is easy without a supporting architecture, nor possible without a basic paradigm.

Export Once it is possible to remove and add parts of a system from and to itself, it is natural to consider moving these parts between different systems (ie, images). Two important examples of this in our current world are the the ability to import a package on SqueakMap into a new release of the system, and the ability to export a project from one system and import it into another. These examples are also important in juxtaposition -- they use completely different mechanisms to deal with boundaries, references across boundaries, and format of the stored modules. Down the road there is reason to hope that these two patterns of use would use the same mechanism.

ImageSegments, Extensions, Install and Uninstall ImageSegments, as many of you know, are a slick piece of squeak technology that allows a well-bounded network of objects to be *very* quickly exported from or imported to, a Squeak image. Well-bounded, here, means not too many in-pointers (ideally only one), and not too many out-pointers. Several years ago, we showed that the entire VM construction category, then about 500k of code, could be imported in a tenth of a second using ImageSegments. However the fact that the system is not already modular has hindered taking real advantage of this technology.

A key issue in partitioning the system is that of "extensions". By these we typically mean, and I specifically mean here, changes to shared classes that are (or should be) local to a package. A typical example would be a String method 'asURL' that makes sure it is a well-formed URL. Such a method would have no place in a system that lacked network support, so it should be a part of the NetworkSupport package rather than part of the base system. This example is fairly compelling, but even seasoned Squeakers are hard-put to say whether some certain utility methods should be included in the base classes or not.

Just a minor nit related to this key issue. It appears to me that you use the word "extension" in its typical meaning: an addition of methods to shared classes, not changes in general. But the same characteristic (changes that are or should be local to a package) applies to a related notion, "overrides" in VW speak. Overrides are needed when you need to modify an already existing method or class definition. Envy for example does not allow them, so when you need a hook in some base class, although you have the notion and support for extensions, and everything else is nicely encapsulated in its own package, you still have to install your hook directly in the other, shared package, modifying it. This has always seemed unclean or incomplete to me, so I welcomed the support for overrides in VW/Store. Plus, if you have overrides, when you unload or uninstall your module, the hook is gone with it. I think it is also important to have this kind of "closure" for our modules from other perspectives, like security: it is a whole module that you trust or not, so if something that conceptually belongs to it lives in a different module, it will have also have different trust characteristics.

Install and Uninstall are the processes required to install a package into, or remove it from, a host system. Typically, installation registers a number of "entry points" or names in various global tables. Our fileIn process already does much of this through its interning of Symbols and handling of global and pool variables. Uninstalling involves reversing these changes and others more problematical, such as the extensions to classes outside the package.

A Couple of Desiderata Much of my recent thinking about modules has been about how to make install and uninstall trivial. I believe it is possible (and desirable) to make an architecture in which different modules can make conflicting extensions to system classes, and in which the *only* thing required to install a module is to read it in as an imageSegment, and the only thing required to uninstall it is to nil the (only) pointer to it. A related desideratum is that, having imported a squeak project, however complex, the *only* thing required is to remove all references to it, after which it will be reclaimed by GC, and no trace of it will remain in the image. When we build a window out of a rectangle, a border width and a color, we just store three pointers; why should it be any harder to combine three modules?

Isn't this just Packages? So far, this represents my thinking of about three years ago (except I was stumped about extensions at that time), and it really reads more like "Future Packages". So the first thing I want to say is that I am in no way critical of all the great ongoing work on packages, both in package design and the effort to split up the Squeak image. Even if we come up with a better architecture for encapsulating modules in memory (and I hope we do), all of the work done so far will be relevant and useful. Moreover, I believe it is the case that the better things are partitioned, the easier it is to move from one module architecture to another.

But Wait, There's More A big shift in my thinking since that time has come from learning about E (see http://www.ERights.org). E is an architecture for secure distributed computing than can be implemented in a number of languages. Its real strengths can only be realized in pointer-safe systems, though, which makes Squeak (potentially) a very attractive host system for this architecture. The essence of E's security is modularity done right. As Mark Miller says, "security is just extreme modularity".

E in less than a Nutshell [There is a fine paper called "E in a Nutshell" by Mark Stiegler http://www.skyhunter.com/marcs/ewalnut.html ] The E work is well thought out and profound. It surrounds a module that they refer to as a vat, which can only be accessed by capabilities. I am not deeply familiar with their implementation, but I think the best way to think of capabilities in Squeak is as a set of message selectors. If a module has its own name space for selectors, and if it is impossible to forge a pointer, then if you have a module pointer and a set of selectors, then the *only* access you can possibly have to that module is to send exactly those messages.

The profundity of the E work extends to distributed computing on multiple hosts with encryption for inter-host communication, and a process model called "promise pipelining" which is like an efficient lazy publish/subscribe. They deal with race conditions, unreliable links and the whole nine yards. Moreover I believe that their model of computation also covers most of the problems of leveraging multiple processors.

A Point of High Leverage I have no delusions that we can pull off an E system in Squeak in the next six months. However my interpretation of what a module is, namely a rigorous boundary in the runtime architecture, is exactly the point at which an architecture can be made to support the requirements of E (or not). The benefits of carrying out this particular piece of work in a manner that is consistent with the requirements of E could be tremendous. For instance, imagine a 64-bit Squeak with Croquet and an airtight security model. That is why E is required reading for this project.

• Dan

On Tue, 22 Feb 2005 15:34:29 -0600, "Ken Causey" ken@kencausey.com said:

...

On Tue, 2005-02-22 at 14:36 +0100, goran.krampe@bluefish.se wrote: ... As you have seen Ken Causey has already picked up the ball and is proposing a Team for tackling one of our biggest challenges - Harvesting. Both in the short and long term. Perfect.

I just want to clarify something regarding this. At least in the short term the team I'm forming (called 'Janitors' for now, send email to janitors-subscribe@discuss.squeakfoundation.org to express preliminary interest in joining the team) is going to be very narrowly focused and will not be replacing or supplanting any existing groups including the Harvesters. Instead we will be working to manually smooth the harvesting path and keep the flow going. Of course longer term goals are much more nebulous at this point. More about this soon.

Understood... the Harvesters are more or less whoever has access to the update stream, the new group wouldn't replace them. But I think the new group would have a lot to say about improving the harvesting process in general. And I assume bug/fix/enh tracking is one of the main focuses of the new group.

- Doug

Ned Konz ned@squeakland.org wrote:

On Tuesday 22 February 2005 5:36 am, goran.krampe@bluefish.se wrote:

...

The overall theme of this Team is to evolve the technology we already have and to make working solutions for the current and near term Squeak releases. And of course to make it all fit together. Without me knowing if people are interested I can *at least* see the following names in this Team:

Doug Way, Lex Spoon, Göran Krampe, Avi Bryant, Alexandre Bergel, Ned Konz

I am not going to venture into talking about who should be Team leader here - I have a favorite though and its not me ;). But I am definitely signing up as a very active member.

Can I help?

Of course. :) We haven't created the list yet - because the moderator should be the Team leader. And the Team leader question is still open. Interested?

In my book you or Doug would fit best.

regards, Göran