Hi all!

NOTE: This email is long but if you care about SqueakMap it should be
worth reading since the responses I get will decide if I want to
continue working on it. It also contains future plans for SM.

I was chatting with Craig today and spurred by Craig's perception that
SM was "file based" I wanted to explain that "No, it isn't really" (some
things are - like the file cache, but most things aren't at all). This
led to a discussion which both explained what SM is and is not, how it
works today and what I want to change in the future. Craig also
explained a few things about Squat. 

Since the domain of discussion is quite complex we probably made circles
around each other :-) but eventually I focused on how I could write an
SMInstaller class that would enable SM to install Squat modules. After
having talked about that - and it seems very doable to me to add such an
installer (though it can be argued to be useless until SM can be
installed into Squat) Craig told me this (for full transcript see
http://tunes.org/~coreyr/read.php?chan=squeak&date=04.07.15):

<Craig>  so, from my point of view, this SM installer for Squat things
adds no value to what was already present in the minimal system, as far
as loading Squat modules goes
<gokr>	Really? 
<Craig>	and everything else SM does seems to support changes that live
in files (fileouts, MCZs, etc.)
<Craig>	so... can you see why I'm not enthusiastic about keeping SM
around for the long term?
<gokr>	So you keep track of people and who maintains the module? And the
co-maintainers? And releases notes, description of the module,
categorization of the modules etc?
<Craig>	yes

So my perception here (and Craig confirmed that) is that Craig is
implementing a replacement of SM, and this I did *not* know about. I
knew he was building a "module system" but since he haven't contacted me
about how it would relate to SM I simply assumed it was at a lower
level, like say Monticello is in most ways.

But the discussion at this point showed me that no - this was indeed
intended as a full replacement and that his perception of SM was that it
was a "transitional" thing. He also said that his stuff is in the
current 1.8alpha release.

(I have looked in there and only found a few unfinished classes -
nothing resembling what SM has today. Remember - SM is *the meta
catalog*, not the package loading/unloading mechanism - that stuff is
handled by MCInstaller, SAR, Monticello, ChangeSets, Project loading etc
etc)

Now, just to make sure I am not misunderstood:

1. Craig can of course build whatever he likes, and I adore the work on
Squat!

2. Craig and I, given the complex subject and IRC as a media, may have
misunderstood each other, even though I doubt it.

3. I am not accusing Craig of anything, even though I *can* say I was
quite a bit surprised and did feel pretty low about the whole situation.
I will not deny that.

I have invested a lot of time in SM and I have been committed to
evolving it into the future. SM has never been viewed by me as a
"transitional thing". How some things are implemented - sure - but not
the system as a whole. I would never put all this energy into a project
if I didn't believe it could evolve into the future.

Anyway, this leads me to the following questions that I need answers to:

- Does the community want me to continue my work on SM? If the answer is
no, I will drop it immediately, I don't have time to waste on a dead
project.

- If the answer is yes, I will equate that to mean that you want SM to
evolve into the future and not just be replaced with something else. For
my future plans, see below.

- If the answer is yes, is it fair of me to ask people to collaborate
with me on how SM works and how it should evolve instead of people
building replacements? If I together with other interested parties can't
evolve SM into the future - then I will be glad to step down and let
someone else do it. But I would like the chance from the community to
evolve SM in the direction it wants to go and I would happily be
convinced about new ways of doing things in SM, like for example the
problem of meta catalog distribution/synchronization, see below.


Future plans in broad strokes
==================================

There are a few cruical parts of the current SM that I dislike:

1. There is no dependency model yet.

2. The synchronization of the model uses HTTP and sends ImageSegment snapshots of the complete SM model down to the client, if it has been modified on the server. This is both crude and forces the model to be exactly the same for all users.

I have just spiced up the SqueakMap Package Loader with some needed FIXes and ENHancements and some refinements to the domain model. My next immediate step is to bring the server a step forward with some FIXes and features and also add the server side cache and a mechanism for mirroring.

After that I intend to add the long planned dependency model. And finally - when that is done - I want to attack the synchronization mechanism adding the following features:

1. The ability to have local differences - like for example private packages that noone else can see.

2. The ability to modify the map using local code instead of going through an ugly web UI on the master. This would open up all the mechanisms in the web UI to local UIs inside Squeak and we maintainers can build and use tools inside the image instead to make releases etc.

This last part - the synchronization mechanism - is tricky.

The ultimate solution would be to have a proper transactional shared object memory like GemStone has. Magma is looking mightily tempting in this regard. Unfortunately it is a pretty hefty dependency to add - but on the OTHER hand - it would only be needed for write access to the map. All the current uses of the package loader in the image would not need it, only if you wanted to do map maintenance it would be needed. Unfortunately Magma doesn't have any mechanisms for doing offline work today if I am correct.

Now - I am all ears about alternative approaches to the synchronization mechanism but remember the following characteristics of the current approach (sending a gzipped ImageSegment through HTTP):

1. It works through any proxy or firewall since it only relies today on HTTP.
2. The map is shared by all other images in the same directory so when you have updated it in one of those images the others will load it from disk instead of wasting precious bandwidth once more.
3. The map works just fine (readonly) even if you are offline.

In fact - the coolest thing would be if someone could build me a mechanism which can take an instance A (of SMSqueakMap) and an instance B and then compare them and figure out the differences. If I had that - then a user could do local modifications to the map - say add a new release of a package - and then "commit" them to the server. It would first compute the differences made by comparing the current modified map with the latest snapshot on disk - and send the differences over the wire to the master server. The master would then either say "No conflicts, committed" or "Conflicts detected - they are these" and the user would have an opportunity to undo selected conflicting changes and try to commit again.

So... I want a smart Smalltalk diff/patch but for *objects* and not Strings. Anyone up to the task? Magma should in essence already have this capability somewhere in there I think. :)

Hmmmm... I might have an idea how to do this in a simple way... Well, that is another posting alltogether. :)

Finally - as I have *always said* - if you are interested in working on SM together with me in any way - email me.

Over and out, Göran


PS. I wrote this in a slightly bad state of mind. If I am upsetting Craig or anyone else, then I am truly sorry. I still want this off my chest and discussed.