Hi!

I visited OOPSLA and spent quite some time with Henrik discussing the
modules system.
I will try to ask a few of your questions, not all perhaps, but a few
that I think I know the answer to. I have intentionally not snipped
stuff here so that others can fill in other answers.

Remember, I may have misunderstood something too - but I hope I haven't.
:-)

"Andrew P. Black" <black at iam.unibe.ch> wrote:
> Dear Henrick, Andreas, and others who may still be interested:
> 
> I'm giving up on this thrust.  It seems clear that I am not the only 
> person with qualms about the way that the module system is 
> developing, but it is also clear that nothing is going to change 
> because of further discussion here.

Hmm, I do not think this is the case. People listen. I talked to Henrik
at great length about a number of issues and he is definitely listening.
Just keep it all in a civil tone. Modules is obviously a touchy subject
- we need to keep our heads calm!

> I was just reading Augustin Mrazik's short paper on structured name 
> spaces, which includes the following statement: "... first the 
> environments and polymorphic class identification have to be designed 
> on a conceptual level, and just then implemented in the system". 
> Well, that is perhaps the conventional  way to proceed: the other is 
> to implement something, and then try to describe it at a conceptual 
> level.  This seems to be what is happening with modules: the only 
> effective description of what a module is and does is in the code.
> 
> I belive in XP, and I've seen XP work well, and XP certainly has the 
> flavour of implementation first, description later, if at all.  But 
> XP has other principles and practices that protect the team while 
> working in this style:
> 
> 	* There is a user story that defines a goal for each coding effort
> 	* There is continual face-to-face communication between the 
> members of the team
> 	* There are tests, both unit tests and customer-defined 
> acceptance tests
> 	* There is a system metaphor, which guides the implementation
> 	* The programmers write the simplest thing that could possibly work.
> 
> The reason that I'm concerned about the current work to implement a 
> module system is that it seems to lack all of these attributes. 
> Which is why I thought it important to come to an agreement on a 
> simple semantic model before implementation started.

I also like XP but nevertheless good code can be produced in a number of
ways. I mean - not all code written before Kent came up with XP is crap,
right? :-)

I have studied Henrik's stuff a bit (a few hours on O'Hare for example)
and I find the code to be of good quality (in my eyes at least). And I
personally do not find the concepts too hard to grasp, but I also
acknowledge that Henrik is trying to be a good OpenSource citizen here
and following Linus' words: "Release early and often."

This way we can discuss and explore together. It also means that there
still is stuff to code. I find that much better than if he just built
something completely without talking to anybody. Well, my 2 cents on
that.

> I think that nevertheless something good will come out of this; there 
> are good implementors working on the module system and it will 
> probably end up serving my purpose.  I'm more concerned that it will 
> be so complex that I never figure out how to use it, which is the way 
> with many "powerful systems".

I do not think you need to be afraid of that. I actually find it
refreshingly NON complex. I mean, hey, the old tools even work ontop of
it! :-)
 
> It's also clear that email does not take the place of face to face 
> conversation.  I've been asking what I though were simple questions, 
> and Henrick needed 12 kB of email to respond -- but never actually 

That should probably be Henrik, not Henrick. ;-)

> answered them.  I'm not expecting you to spend more time answering 
> this, but I hope that you will take the opportunity to come to Bern 
> and visit with Stephane and myself, and see if we can't bridge the 
> semantic gulf that seems to separate us.

Straight talk (being frank, but do not intend to pick a flamewar):

I do know for a fact that Henrik really tries to answer/discuss it and
be a good "citizen" here.
He also likes all the questions which only mean that we care.

But I (I do NOT talk for Henrik here) have reacted a bit about the
"tone" of this discussion. I mean, hey, Henrik has coded up a really
good chunk of OpenSource code! I find it a bit "below the belt" to
question his particular development method since we do not have any
insight to do so. We only have the code. Judge that on it's own merits.

So, back to the code:
 
> Henrik writes:
> 
> 
> >You are also right in that the differences between regular and delta modules
> >should be as small as possible, and that we should be specific about what
> >they are. However, the two cannot be identical: a DM needs to be defined
> >relative to a base module, and the concept of a base module is meaningless
> >for a regular module. This ought to be the only real difference. Then we may
> >want to have some tool-like support for viewing DMs in terms of differences
> >wrt the base module, but those are minor issues.
> 
> Can't any regular module be defined as a Delta with respect to the 
> null (empty) module?  Then the empty module could be the only regular 
> module in the system.   An analogy: this is like defining all lists 
> as the catenation of elements on to the front of a single empty list.

Interesting thought. But does it lead anywhere?

A deltamodule is in a sense "just a diff" between two different modules.
(typically we would think of two different VERSIONS of one module - but
it does not matter)

This means that given a module A and the deltamodule to turn module A
into module B you should be able to "apply the diff" and get module B.

Similarly you should in theory be able to produce the same deltamodule
given just module A and B.

Of course - in theory, the deltamodule from "the empty module" to module
A would probably be in some sense "info-mathematically" equivalent to
the module A.

In analogy this is the same to say that a ChangeSet which creates 10 new
classes is actually the same as a fileout containing those exact
classes. Yes... but they are still different beasts right? The changeset
contains changes and the fileout contains "a state".
 
> >I think that this situation between Modules and DeltaModules is rather
> >common in class/subclass relations, and I think your question can be
> >understood by analogy to other known cases: why aren't the protocols for
> >Array and OrderedCollection identical, and so on. It is because they are
> >very similar but still slightly different.
> 
> The protocols for Array and OrderedCollection are not identical 
> because the reason for the existence of OrderedCollection is to add 
> some functionality, specifically, the add: and remove: protocols. 
> This seemed completely different from your description of the 
> difference between Delta Modules and regular modules, which was that 
> they had a different implementation.  That seemed more like the 
> difference between a Stream writing to a file vs a Stream writing to 
> an OrderedCollection: the implementation is different, but the 
> protocol ought to be the same.
> 
> 
> >The result of activating/installing a DM is equivalent to creating a
> >new/different version of the base module by filing in all the contents as a
> >regular Module. (I now think that de/activating a DM is a better term than
> >un/installing it.) You aren't forced to use DMs for class extensions, but
> >filing in five new methods is clearly more convenient than loading a new
> >version of the whole module that contains String.
> 
> If filing-in is the mechanism for loading functionality, that is 
> certainly true.  In VisualWorks, of course, they load something like 
> image segments, and the user can't see the pause.

The exact format of the modules "on disk" is changeable I think. If
someone would like to code up support for SIF or Segments or whatever
then we could just dive in. Currently I think Henrik uses a chunk-format
of some "familiar-looking" kind.

> >>  I just put my "loose
> >>  methods" (like String >> asUrl) into a DeltaModule inside my Module,
> >>  and the unsuspecting user who imports my module is still surprised
> >>  that it "damages" other classes like String, or Form, or whatever.
> >
> >I don't agree at all. DMs not only collect all external modifications in a
> >special place, instead of mixing it with the "proper" contents of your
> >module. They also associate such modifications with the modules where they
> >are made, so that you by looking at the list of delta modules can see
> >exactly what parts of the system a certain package modifies. Any
> >"unsuspecting user" would most probably look at what parts of the system a
> >module modifies before they use it. In this way this is made easy. Can you
> >suggest any way to make it clearer?
> 
> Yes, I can suggest a way to make it clearer.  It would be clearer to 
> have a module browser tool that examines a module and tells me what 
> the module does -- what classes it adds,and what classes it changes 
> -- before I load it.  If the module that I'm browsing contains 

Ok, now I can see a problem in communication here. At least at OOPSLA
Henrik made it quite clear that we have two stages here:

loading/unloading followed by activating/deactivating

Just loading a module into Squeak does not activate it. This means you
can load any module (delta or regular with all contained submodules)
into Squeak and then browse them etc. Thus you can see exactly what a
module contains before you activate it - especially all the deltamodules
it contains. If it has no deltamodules then it is in fact "just a bunch
of globals/classes" and can not conflict with anything else.

Then you choose to activate a module and that will most importantly
install the deltamodules that comes with, it typically replacing/adding
methods in classes from other modules.

And then when you deactivate it it will undo that and bring those
"touched" classes back to where they were before.

Well, did I make anything clearer? :-)

> sub-modules, I can see their structure too.  I can also selectively 
> turn off (i.e., elect not to install) any of the components.  Think 
> of a Mac installer, where you can do the "Easy Install", or you can 
> go to a Custom Install option that lets you check boxes for each of 
> the sub-components that you want, and gives you information about 
> each of them.
>
> The reason that I would prefer such a tool is that it would tell me 
> what is _actually_ in the module, not what some programmer said was 
> in it before the last round of changes!  It doesn't rely on 
> conventions, like "additions go only in Delta Modules".  Hence,  I 
> can't be confounded because someone doesn't observe the conventions.

Eh, first of all - as I outlined it above this kind of tool definitely
fits in with Henrik's code!
It just isn't written yet. But you can do exactly what you describe
today with code and a slightly jazzed up objectexplorer.

And secondly - the delta stuff is definitely not "just a convention".
Since a regular module can not contain a "loose method" for example it
MUST be put in a deltamodule. Perhaps you misunderstood Henrik when he
wrote:
 "You aren't forced to use DMs for class extensions, but
  filing in five new methods is clearly more convenient than loading a
new
  version of the whole module that contains String."

I think Henrik meant that instead of using deltas you could in fact
create new versions of the base modules that the deltas modify. In
theory yes. But hey, who would like to ship a new special version of
Morphic every time you deliver something! :-)

For example, the way it will work (very shortly I think) is that when
you work on a Project (which as default will have a Module which
collects all new classes you create in that Project (like the current
ChangeSet)) and suddenly add a method to Object, Squeak will produce a
deltamodule (on the module that has class Object) within your Module
containing that change.

Of course, it could confirm this with a dialog or whatever if you have
that preference set. And then when you add a method to Morph it will
create another deltamodule (since Morph clearly does not reside in the
same module as Object) in your module. So now your module contains two
little deltamodules.

Now, if you deactivate your Module (switching Project perhaps I guess)
the module system will deactivate those two deltas.

So you could just hack away like you have always done (if preference is
set to "do not ask" or something) and later you can take a look at your
module and discover a bunch of deltamodules. Aha! Perhaps something that
you need to take care of? Or it might be just what you want.

> >This also serves to make it very clear to a conflict resolver in what parts
> >of the system conflicts arise with other loaded packages: conflicts need
> >only be handled in those modules where multiple sources want to have
> >modified versions of the same module.
> >
> >>  Now, I had assumed up until this morning that a Module could contain
> >>  not only whole classes, but also "class extensions", that is, groups
> >>  of methods that could be added to existing classes, even though those
> >>  classes might be defined in different modules.  (The String>>asUrl
> >>  example again, which I had assumed could be part of the HTML module).
> >
> >But the very purpose of modularity is to separate a system into smaller,
> >independent, self-contained pieces, whose definitions are not entangled in
> >each other. Modules really should not be allowed to modify the contents of
> >other modules. Allowing this would go against the very idea of
> >modularity--it is really a contradition in terms, more or less. It is the
> >same violation as if an object would be allowed to directly alter the
> >innards of another object, it breaks encapsulation.
> 
> So are you saying that Modules cannot modify the contents of other 
> Modules, or that they can?  I'm confused again.  Is a Delta Module a 
> Module, or not?  It sure sounded like a module up until now.

Ok. Lets see know. A Module can only contain classes (well, actually
globals but let's keep it simple). A "loose method" can only be kept in
a deltamodule.

BUT... since a module CONTAINT submodules (Modules or Deltamodules) a
regular Module typically contains a bunch of Deltamodules that define
all the changes that needs to be done to other modules in order for it
to play nice in the image. So your Module is actually a tree with
potentially both Modules and Deltamodules within it.

If a Module does not have ANY deltamodules anywhere in its
submodule-tree then it will not modify any other modules upon
activation. I personally find this rather neat and tidy! :-)

> >>>  A module
> >>>  * will of course also have contents proper: classes, globals, etc.
> >>
> >>  I notice that "proper contents" does not include methods.  Is this
> >>  intentional?  Is this a mechanism restriction that is intended to
> >>  make it hard for me to put the "wrong stuff" in my module, that is,
> >>  to put in "loose methods" that change some other classes?
> >...
> >>  If so, I admire
> >>  your good intentions, but I think that you are misguided.
> >
> >This is just because in ordinary Smalltalk you don't put methods anywhere
> >except as part of classes, this just follows that principle. You may
> >consider Smalltalk to be misguided when it enforces this principle, and
> >would then regard C++ as superior in this respect, but many would consider
> >this part of the essence and elegance of Smalltalk.  #Smalltalk contains
> >classes and other globals, but not loose methods, it's the same thing with
> >modules. In fact, neither DMs contain loose methods, they are put in class
> >objects too, at least as it is now. In fact it would be more work to support
> >the kind of exception you suggest, than to simply put methods in classes
> >like now.
> 
> 
> What I meant was not that there could be methods that do not belong 
> to any object, but that when I am programming I can not only create 
> new classes, but also add methods to existing classes.  This is one 
> of the strengths of Smalltalk, because it lets me put the methods in 
> the right place.
> 
> I think that you are saying that Regular modules can _only_ define 
> globals, whole classes, and DelatModules, and that additions and 
> deletions to classes can only appear in DeltaModules, either nested 
> within a Module or not.  Is that right?

Yup! I think so. :-)

> >  > Now I see that I might be wrong about this, and that Andreas' point
> >  > of view would say that Modules shall contain _only_ whole classes,
> >>  and DeltaModules shall contain _only_ class extensions and perhaps
> >>  class retractions (deleting methods).
> >
> >Since DMs are defined as differences in relation to a base module, per this
> >definition (and for superseding change sets) they strictly ought to be able
> >to define any possible differences in a module, however some changes are
> >harder to support than others. Still, adding new classes should definitely
> >be possible, and it is easy to support (DMs being Modules, I think they
> >already do). But this would probably be used when DMs are used in the role
> >of change sets--a package will rarely need new classes to be added to other
> >modules.
> 
> So DeltaModules can define globals and whole classes too?  Was I 
> wrong again?  I'm frustrated that after all of this discussion, I 
> still cannot seem to get a three line summary of what things go in 
> which kind of Module.

Eh, well since a Deltamodule defines a difference between two modules it
should be able to contain a whole new class. But you would probably
rather seldomly have one Module add classes to another module so it will
probably be a rare case.

The threeliner (well, we can all try to revise this one so that it gets
really good):

A Module contains globals including classes.
A Deltamodule contains all possible differences between two Modules.
(changed methods, added methods, added classes, removed methods or
classes etc.)
A Module can contain other Modules and Deltamodules creating a tree.

I do not think a Deltamodule can contain other submodules so I think
Deltamodules can only constitute leafs in the tree... Darn, don't have
the code available, I can check tomorrow.

> >>  The email of yours in reply to Michael Rueger also said a lot of
> >>  interesting things about modules being namespaces (not stated on the
> >>  Wiki) and that these namespaces do not follow the hierarchic
> >>  structure of the module nesting (not on the Wiki either).
> >
> >In both cases, the information on how names are looked up was in the first
> >posting I made, and was copied into the first heading listed on the Swiki,
> >"design principles". However, Smalltalk is the only language I can think of
> >where different modules share a single namespace, so I may not have made it
> >as clear, taking it for granted. Again, it seems to go against the very
> >principle of modularity, where modules' contents should be independent per
> >definition. So I've been taking this feature for granted whereas all
> >Smalltalkers obviously don't.
> 
> I felt really bad that I had missed something obvious.   I went back 
> and read it again.  I searched on Namespace.  I still don't see where 
> you describe this at all.
> 
> I did find the following paragraph under "Module Structure" 
> (http://minnow.cc.gatech.edu/squeak/2069)
> 
> >Instead of a single path of inheritance for the names available in a 
> >Module, it should be possible to delegate name lookup to any 
> >(possibly several) imported module or submodule, by marking those 
> >modules for delegation. (The present scheme is equivalent to how 
> >Self replaces s.i. with its multiple inheritance/delegation scheme, 
> >where submodules correspond to slots and delegation corresponds to 
> >marking a self slot as a parent, except in that you can delegate 
> >also to imported modules.) Simple conflict resolution can be done by 
> >specifying an ordered lookup of submodules and imports (in the order 
> >of declaration).
> 
> Is that what you are referring to?  i read this as describing nested 
> scopes, like Algol 60 blocks.
> 
> >DMs should in principle be able to handle all possible differences wrt a
> >base module: new items, deletions, and changes.
> >
> >>  *  BasicModules can contain (sub) BasicModules, global specifications
> >  > and Class specifications, but not DeltaModules.
> >
> >Regular Modules can be linked to any other Module, regardless of their kind
> >(of course). PackageModules etc. are unnecessary.
> 
> I know that PackageModules are unnecessary.  RegualrModules are 
> unnecessary too, since DeltaModules can do anything that a Module can 
> do, and more besides.  The suggestion of adding PackageModules was 
> based on the Andreas' idea that we want a different implementation 
> concept for every possible usage.  We have so far identified three 
> usages:
> 
> 	(1) defining new stuff (we do this in a regular module)
> 	(2) changing old stuff (we do this in a DeltaModule)
> 	(3) packaging up a subsystem for distribution, which will 
> usually require some of (1) and some of (2).
> 
> We could do (3) with a RegularModule, but that would destroy the 
> purity of RegularModules.  We could do (3) with a DeltaModule,  but 
> that would destroy the purity of DeltaModules.   We could  forget the 
> idea of having different kinds of module, and instead have a tool 
> that tells us what a module would do when loaded.  (My suggestion 
> above.)  But if we reject all of these alternatives, then I think the 
> consequence is that we need a third kind of module, the 
> PackageModule, that just does Packaging, and nothing else.
> 
> Do a good job with the implementation, and I'm sure that many people 
> will be happy.

Ok, there might be changed stuff here perhaps - I will leave it to
Henrik to answer. I am not sure I saw anything regarding BasicModules at
OOPSLA. And I did get the impression that the Module hierarchy actually
is a namespace hierarchy.

I will try to take a better look at the image Henrik handed me at OOPSLA
and see if I can come up with more answers.

> 	Regards,
> 
> 		Andrew

regards, Göran