local SqueakMap's (was Re: Feedback)

[goran.krampe at bluefish.se]

Hi Lex!

"Lex Spoon" <lex at cc.gatech.edu> wrote:
> In general, Goran, this sounds like a plan that can work, if you want to
> do all that work!  Let's just keep our eyes on the list of desiderata

Well, I have intended something like this for a long time, I just have
postponed it. :)

> (local development, dependencies, and whatever else), but if they are
> all met that is wonderful. 
>
> Have you thought about mixin universes?  Like, someone backports
> internationalization to 3.7 and sets up a local SqueakMap to hold the
> packages in it, while someone else creates a cryptography SqueakMap that
> holds code that can't be used or redistributed by Americans.  Some
> images might want to have 3.7 + m18n + cryptography, and others might
> want 3.7 + cryptography, etc.  For various reasons, these mixin
> repositories do seem to be useful in Debian, and so presumably they will
> be useful for us too if we support it.  With the design strategy you
> describe, maybe this could be supported by having multiple parent
> pointers?

Yeah, I have thought about them - and yes, possibly. I do think that
would make things more complex though - but at the moment I haven't had
my morning coffee so I will simply ignore it. :)

Perhaps we could postpone mixins until we have the rest working? With
the intention of trying to support them, but waiting until we have the
rest operating I mean.

> > I agree, one design I have somewhere in the back of my head goes like
> > this:
> 
> Sounds good to me, not that I know SqueakMap internals all that well....

The internals in SM aren't that magical. :)

> > 	- When you "commit" your changes you simply send your full SMAccount to
> > a selected server. This means you make your modifications (creating
> > releases, whatever) in your own image and then using a simple HTTP post
> > or whatever, you smack your full account up on the server.
> 
> So an account is the unit of interchange between servers.  Well, I have

Yes, I know it has its problems - it may be tempting to go more fine
granular - but on the other hand that is the granularity of ownership,
so it is natural. You own your account and all in there - it is like a
document kinda. I know it has issues as you point out, see below...

> no idea if that will work.  If it does, terrific!  Does this include
> support for the same account posting packages to a private server?  You
> don't want to upload the local packages to the public server, but you
> may want to reuse the account.  Alternatively, it may be reasonable to
> make people have different accounts on different servers.

I know - first I thought "ouch" and that hey, they will just have to use
separate accounts - but that hurts when you start thinking about it. An
alternative is this:

In essence an SMAccount has a couple of fields and then a collection of
SMObjects. That is more or less all to it. Then SMObject of course have
subclasses like SMPackage etc. One idea is to mark each SMObject with a
"home server" tag. And when you commit your account you need to specify
to which servers (one or more). Then it:

	1. Create a copy of the account disconnected from the map (replacing
all forreign refs with UUID proxies).
	2. For each server to commit to:
		2.1 Deepcopy the copy.
		2.2 Filter the list of SMObjects based on the server to commit to.
		2.3 Serialize the account using ReferenceStream and friends, gzip and
POST.

...this way you would only have one account no matter how many servers
you participate on.

> Incidentally, I organized Universes so that HTTP is used for retrieving
> copies of the universe, but a maintained TCP connection is used for
> editing.  This way, it opens the door to negotiations, as Craig would
> say, between the editing image and the server.

Just a note regarding networking, I have always thought SM should
continue to use simple HTTP in order to always work behind firewalls etc
- but one really nice thing lately is Cees' p2p stuff. I don't know how
it can affect us (SM, Universes) but it sure does have tons of
interesting mechanisms. :)

> But if HTTP POST's are enough, you don't need that.  I prefered to avoid
> having to constrain the protocol that way.
> 
> > Anyway, something like the above seems pretty doable.
> 
> Cool!
> 
> > > Being disconnected does change the design problems in one way: unique
> > > identifiers can no longer be guaranteed.  One general strategy for
> > > approaching this, is that if the local guy defines an identifier
> > > identical to what the global server has, then the local guy is
> > > *overriding* the global server -- just like with class inheritance.
> > 
> > Well, I think the model don't need that - all SMObjects have UUIDs, but
> > sure, there are a few ways to "clash" today, for example - package name
> > is verified to be unique. I was thinking that any such conflicts should
> > be noticed at "commit time" and if there is a conflict we simply abort
> > and tell the user.
> 
> Clashes seem like a fine strategies as well.  Notice that they also seem
> to happen whenever you do an "update" from a more public server; changes
> that have been committed locally, may turn out to clash with something
> that happened upstream.

Well, in general systems yes :) but in SM, no, not really. Because
locally you will only modify your own account - modifying other parts of
the map is a general "no, no". If you do that, then you are on your own
IMHO. :)

>  Going the clashes route, it seems like the
> model now needs to have support for "stuff killed off by an update from
> upstream", plus some way for the local users to fix up the situation.

Yes. Typical issues when committing are:

	- You created a package (or some other artifact with unique names) with
a name that already exists.
	- Your SMObjects refer to other SMObjects that do not exist anymore
(categories could be removed, package releases also etc, etc)

...but without giving it tons of thought I do think the above two things
are mostly it.
 
> I guess another general approach, is to try and rename the UUID's
> locally, in order to undo the conflict.  I don't know if this is doable
> or not.

Hmmm. Not sure I grok that. :)

> And (heck) a fourth, is that you change from UUID's to server-unique
> ID's.  So when you refer to an ID, you have to know which server it
> pertains to.

Well, I might have jumbled up my thoughts - but given the above design -
the SMAccount that you are committing to a server will only contain
SMObjects that think they "belong" there so all other SMObjects they
refer to should be available on that server too.

Which means that, perhaps I didn't write it, but the server marker in
each SMObject makes sure it has a "home server" in the server "tree",
and it will only be able to refer to other SMObjects available on that
server (which includes all servers above it in the tree all the way up
to the master server on sqf). That restriction would be quite easy to
maintain in the client image when you are modifying your account (since
all SMObjects know their home).

So again, the tree of servers with each node down the tree only adding
SMObjects makes these things simpler. And yes, mixins would probably
make thing more complex :), not sure.

One way of making mixins simpler might be to restrict mixins to be a
leaf thing. I mean, the servers are all in a tree like I describe but
the leaf (localhost) could possibly use mixins and take the heat of that
locally. But again, I would gladly wait with mixins until later.

> -Lex

regards, Göran