Hi Doug and all!

Doug Way <dway at riskmetrics.com> wrote:
> 
> Göran Hultgren wrote:
> > 
> > Hi all!
> > 
> > Quoting Doug Way <dway at riskmetrics.com>:
> > [SNIP]
> > > DeltaModules, on the other hand, can contain method additions, but also
> > > changes to existing methods and method removals too.
> > 
> > And in fact any other possible change too (just so that noone misunderstands
> > that, I know you know this).
> 
> Right, I forgot to mention that.  DeltaModules can also add/remove instance variables, etc.

Since people seem to have a problem grasping DMs I just wanted to be
picky. ;-)

> > > Let's think about a couple of useful properties of modules:
> > >
> > > 1. Can they be cleanly unloaded?
> > > 2. Are they behavior-preserving?  In other words, if you load (and
> > > activate) a module, will the rest of the system still run without
> > > breaking or behaving differently?
> > >
> > > [SNIP]
> > > 
> > > But the interesting thing is, Envy/Ginsu-style modules w/Class
> > > Extensions *are* behavior-preserving, with a few caveats* (see bottom of
> > > message).  This is because, in general, if you add a new method to an
> > > existing class, none of the rest of the system would ever send that new
> > > method, so the behavior of the rest of the system is unchanged.  So,
> > > reusing the example above, if a WebBrowser module has a Class Extension
> > > which adds the method #asEncodedHtml (but does not change any method) in
> > > class String, in the module "Kernel 2.7", the Kernel module will still
> > > remain 2.7.  (This is exactly what Envy does.  I don't remember now if
> > > Ginsu supports version numbers.)
> > 
> > Yes, a very crucial observation. Just a note though:
> > You listed a few caveats but there is one more - the new behaviour may of course
> > affect the state of the instance so that other "modules" are affected. But that
> > is probably more of a "bug" than a "source code conflict". Whatever.
> 
> Hmm, I'm not sure exactly what you mean here... could you provide an example?
> 
> Aside from the caveats I mentioned, adding method extensions should not affect the behavior of other modules.  For example, let's say we have a system made up of modules A, B and C, where module A is a prerequisite of B and C.  If we also load module D which has Class Extensions on classes in module A, the code in the original system (A B and C) should still work just as it did before (aside from the caveats), because it won't have any references to the newly added methods.  (Thus, we don't have to update the version number of module A.)

What I meant is that when module D (adding methods to classes in A) is
running it will call the methods in question and that can affect the
state of those instances. If those same instances are used by modules A,
B or C then those modules will be affected. In short - there is no
memory protection between the modules. So if my methods introduce
instance states that are faulty or at least unknown to modules A, B and
C then those modules can very well break.

So we can't be SURE that modules A, B and C are unaffected. But we can
of course say "shit happens". :-)

> > > One last thing: As far as splitting up the current Squeak image into
> > > modules goes, my hunch is that it would be easier to do with
> > > Envy/Ginsu-style modules.  This is because you would not need the
> > > DeltaModule capability of specifying changed/removed methods when
> > > divvying things up, only the ability to specify added methods, which the
> > > lighter-weight Class Extensions can also handle without version number
> > > worries.  For example, String>>asMorph could be quickly dumped into some
> > > Morphic module.
> > 
> > There might also be another problem lurking with DMs - at least when trying to
> > use them to chop up the image. Since a DM branches off the affected base Module
> > then it would seem to me that we will end up with tons of "branches" all over
> > the place. Module C has a DM which branches off Kernel to 2.8 and Module D has
> > another DM that branches off Kernel to 2.9 - oops...
> 
> Yes, this is exactly the problem I was thinking of when I said "version number worries".  This seems like it could be a major problem.  I'm not sure how things like String>>asMorph were intended to be dealt with when the image is partitioned... perhaps most of them would need to be removed completely, requiring a lot of refactoring, so that we don't have branches all over the place?
> 
> Envy/Ginsu-style modules don't have this particular problem, because the additive-only Class Extensions don't require the base module to "branch" (change its version number).

Exactly.

> > [SNIP]
> > The following are a few unstructured thoughts:
> > 
> > Currently I think that producing Envy/Ginsu style packages (for example using
> > DVS) consisting of standalone classes and loose method extensions will work
> > quite fine for many of us (as you also noted) - especially if you build stuff
> > "for" 3.2, compared to trying to detangle 3.2 itself...
> > 
> > But I would like to have these "loose methods" contained within a new smarter
> > kind of ChangeSet that I will call a "Patch" here below. I will try to explain
> > why and what such a beast would be.
> > 
> > - A Patch would be similar to a DM in that it is not a recording of a sequence
> > of changes - it is instead a condensed set of changes, much like a patch-file in
> > Unix.
> > 
> > - On the other hand a Patch would not be a Module (as DMs are) or anything like
> > that in itself. Instead a Patch is an "intermediate object" like DVS and other
> > tools could produce when they want to represent "loose method additions" or
> > "loose method removals" or "class instvar additions" etc. Patches would be able
> > to represent themselves in a simple format that can be used inside fileouts and
> > of course they would be instantiated when filing in such fileouts.
> > 
> > So the Patch would not live in the image like classes do but tools could produce
> > them as an intermediate object when filing in or out. And when installing them
> > it may be a good idea to keep them around.
> > 
> > Ok. A Patch would not try to replace ChangeSets in any way. A ChangeSet is a
> > "recorded session" in some sense and thus IMHO more of a development tool than a
> > packaging/deployment tool. I think we should keep using them for source code
> > management but perhaps we should try to avoid using them for
> > deployment/packaging and instead rely on .st files containing full classes and
> > Patches.
> > 
> > Here are a few crucial points... A Patch object should:
> > 
> > 1. Keep enough state about the changes so that it can be used to detect
> > conflicts against other Patch objects and packages. For example: One Patch
> > adding a loose method x in class A would not conflict with another Patch adding
> > method y in class A - they would both be additions. A Patch changing a method x
> > in class A (belonging to package AA) would on the other hand potentially
> > conflict with a package BB that depends on package AA. But perhaps we could even
> > add classifications of "method changes" - one being "no behaviour change"
> > (changing comments, formatting, optimization etc) and then it could be resolved
> > into a "theoretical conflict" (likely not a conflict but could be if the method
> > change is actually changing behaviour nevertheless) instead of a "conflict". So
> > in short - we can arm these Patch objects with a lot of "smarts" for detecting
> > conflicts and also help us resolving them.
> 
> This could be interesting.  (The "no behavior change" message classification sounds neat... renaming a temporary variable might be another example.)

Right.

> > 2. Keep enough state for deinstallation. This part is (as I have argued in other
> > posts) not truly critical but if we try to implement these little fellows we
> > could at least give it a stab. It would be nice.
> > 
> > Both of the above features would need the Patch objects to stay around after
> > filing in, not necessarily in the image though - we could log them in a file or
> > something.
> > 
> > So in short, the proposal of a lightweight module system could be like this:
> > 
> > 1. We implement the Patch object as described above.
> > 2. Avi :-) incorporates Patch objects into DVS.
> > 3. We start using .st files with Patches inside them for packaging/deployment.
> > One way would be by using Envy/Ginsu style packages with DVS.
> > 4. We add dependencies to SqueakMap between packages. We would also need to add
> > versions of packages of course because currently SqueakMap only knows about the
> > latest release).
> > 
> > In Henrik's Modules dependencies were declared "per Module". Perhaps that level
> > is too fine grained? If we add dependencies in SM it would be "per package"
> > which is a larger and perhaps more appropriate level.
> > 
> > All this would mean that we can continue using ChangeSets if we like. Using them
> > for deployment would lead to "uninstallable" packages with much less capability
> > of conflict resolution.
> > 
> > Then when we try to install a package using SM it holds information on what
> > packages we already have installed. Packages using .cs or "interactive"
> > .st-files can't really participate in the conflict detection - hopefully those
> > kind of packages will "phase out". But the "new" .st format with Patches
> > complemented with dependency information in SM should be both uninstallable and
> > especially conflict detection between them should work nicely.
> > 
> > In short: The new Patch object is IMHO the key for successful packages/modules.
> > And the simplest Patch is a "loose method addition" which will give us
> > Envy/Ginsu style modules. But then nothing stops us from evolving the Patch
> > concept with more smarts.
> 
> This type of conflict resolution could be very handy.  I don't have time at the moment to comment more on it... and I should probably think about it some more first anyway. :)
> 
> In any case, I suppose one lesson learned is that there are many ways to implement a modules system in Smalltalk.  I was going to try to summarize/categorize them, but that seems like a task for another time. :)  And it may be that some of the problems I talk about above could be addressed by Henrik's module system with some tweaks... I guess I wouldn't rule anything out yet.

True. After having described the above ideas with Patch etc I also
(don't remember if I posted or just emailed Daniel about it) realized
that a very simple way to achieve something similar is to start
classifying updates ChangeSets. If we introduce update queues for
packages in SM (which seems pretty high up on the todolist right now) we
could use the category system in SM to classify those updates much like
I described above (no behaviour change, should be backwards compatible,
not backwards compatible etc) and then we could get "stretchy
dependencies" from that. 

If I didn't introduce that term it means more or less that if a package
A is dependent on version 1.22 of B then if B moves to 1.25 by 3 update
ChangeSets all classified as "no behaviour change" then we can be sure
that A will work with B 1.25 too. And if say the last one is classified
as "should be backwards compatible" we can still be pretty safe, etc.
So, the dependency can "stretch".

> - Doug Way
>   dway at riskmetrics.com

regards, Göran