The future of SM...

goran.krampe at bluefish.se goran.krampe at bluefish.se
Mon Jul 26 11:58:33 UTC 2004


Hi Lex!

WARNING: This is a humongous posting. If you are not really into SM and
dependencies, the question on SM architecture etc - it is probably DEAD
BORING.

(ok, taking time to answer as much as I can, coding will just have to
wait)

lex at cc.gatech.edu wrote:
> Hey Goran, I truly have been reading what you and others post.  For

Ok, then. It seemed to me that you hadn't.

> example, I showed that Julian's "problems" are actually easier to solve
> with multiple maps, and no one responded.  Also, I truly am familiar

Can you point me to that particular post? Or perhaps it was the one I
responded to in this posting (the walk through).

> with the details of SqueakMap that you are describing, though I have not
> delved into the details of its implementation.  Please do take what I
> say seriously; I am not just covering my ears and harping my favorite
> tune.

I *am* trying to take it seriously.

> There are two main desiderata I mentioned that you say you are not
> planning to implement: simple unversioned dependencies, and allowing
> clients to pull from multiple maps.

"to implement simple unversioned dependencies". Well, that depends on
what you mean - I don't intend to *record* such dependencies *but* they
are of course *implicit* since I store more information. (note: a little
bit more on this below, where I talk about "intended" dependencies)

"Allowing clients to pull from multiple maps". It is not clear to me
what that exactly means.

I have explained that I will move towards a tree (up side down, well
that is my inner picture of it) structure of servers where the master is
the root and where you can connect to a chosen server down the tree thus
getting access to SMObjects not visible higher up (closer to the root).

Also, note that the map is a catalog - it is not a remote directory full
of package files.

IMHO the mess that Debian suffers from because of the simple fact that
they don't have a central catalog is not something we want. If
information is meant for all of us - then it should easily be available
to all of us - and I don't want to worry about if there is a package
somewhere on some server that I don't know about.

> On dependencies, I would love if you were to actually step through
> either the example I posted or one of your own. Any example will do so
> long as it is not in lockstep and so long as you do not require
> interaction with the user to make the decisions.  What do you envision
> the package loader will do at each step?
> 
> 	http://lists.squeakfoundation.org/pipermail/squeak-dev/2004-July/080076.html

Sure. In fact I am replying to it all "in place" here:
--------------------------
> Individual authors are not in a position to make guarantees about
> releases, regardless of the technology.  Arbitrary combinations of
> packages can interfere with each other.  Additionally, there are no
> guarantees in software engineering to begin with.  Tomorrow the sun will
> rise again and someone will find another bug in my code.

Eh? Guarantees? I wrote "promise" and what I of course mean is that the
maintainer tells me that, sure "my package X0.09 works if you have Y1.2
and Z1.3" installed, because that is what he has tested.

Of course it isn't a "guarantee" - but it sure if much more precise than
saying "my package X works if you have Y and Z".

So I have no idea where you are going with arguing about "guarantees" -
we both know what I am saying.

> If you really think there are guarantees to be made, though, let me give
> you two examples to consider:
> 
> 	1. How do dependencies keep someone from changing the behavior of #new
> to automatically call initialize and thus break your code?  Does every
> package have a dependency on every part of the system kernel that it
> uses?

Of course not - don't be silly.

> 	2. How do you make a guarantee of correctness, when some other package
> might modify the "open" menu in away that crashes the system whenever
> the "open" menu is used?
> 
> And if you *still* like "guarantees", how do you fit post-release bug
> reports into the world view?  I guaranteed it yesterday, but now a new
> bug has been found; do I un-guarantee the package?  What is the point of
> a guarantee that can be un-guaranteed later?  Maybe I should just fix
> the bug and post an updated package and stop wrangling with
> dependencies.  :)

You are rambling and if you keep arguing in this fashion then I will not
bother discussing this. I haven't said "guarantee" - and even if I had
used that word it was bloody obvious what I meant.

I as a maintainer can only say that "Yes, my package release 0.09 works
as far as I - as the developer - can say, given that you have package
releases Y1.2 and Z1.3 installed that it depends on. If you have older
releases - I don't know, because I haven't tested those yet. If you have
newer releases, I don't know - I especially don't know about *future*
releases because I have no idea what will change in Z or Y tomorrow".

Btw, you did stumble upon a slightly interesting fact though - you said
"how do you fit post-release bug reports into the world view?". Given
that in the planned SM the dependency information is not embedded in the
actual release (as in Debian) this means that the dependency information
can be revised post-release. And it should, especially by adding more
working know configurations. If the maintainer or anyone else discovers
that, darn - there was actually a serious bug in Z1.2 and Z1.3 is really
needed - then the configuration can and SHOULD be changed.

And if someone discovers that, hey "I used Z2.0 and it seems to work
fine with that too!" then that should/could be added too. That is one of
the key points of NOT embedding dependency information inside the
releases.

> Aside from the issues with the "guarantees" idea, it has not been
> established that the jigsaw puzzle of version-specific dependencies is
> going to be practical for tools to work with.  This is a significant
> technical problem that versioned dependencies add, and everyone seem to
> be waving it off as something to handle in the future.  I don't know
> that this is solvable, guys, and no one has made a convincing argument
> otherwise.

Well, I haven't built this yet - so how can I be sure. But I have
thought about it and I have discussed it with many people. Now I want to
give it a shot and we can see. If I fail - so what?

Then you can gloat and laugh and make a fool out of me as much as you
like and then take over maintaining SM. ;)

> In addition to those two problems, there are definitely times when the

Noting that the first "problem" was... well, what was it? Something
about me using the word promise. And the second was that... we don't
know if it will work until we have written it? Sure. Like all software.

> dependency jigsaw cannot be adequately solved.  Let me run you through a
> small example.  Packages A and B depend on Collections:
> 	
> 	A 1.0   needs   Collections 1.0
> 	B 1.0   needs   Collections 1.0
> 
> Fine so far, I install all three packages.  Now the collections library
> gets updated to 1.1.  I cannot install the upgrade without breaking the
> dependencies of A and B!!  So, I wait before installing it, even though

Not entirely true. First of all - the engine should be parameterizable
(cool word) by you.

So the idea is that if you - at this point - select Collections 1.1 and
press "install" it could say something like "The installed package
releases A1.0 and B1.0 are only known to work with Collections1.0, but
since Collections1.1 is marked by the maintainer as being 'Code changes,
but only big fixes' they could still likely work, do you still want to
proceed and install it?"

And if you set a parameter, let me fantasize here - "Allowed
compatibility threshold" or something - then perhaps it wouldn't even
ask if the new release was below a certain compatibility level.

Now, it doesn't end there. Not only can you install Collections1.1 (and
as I described in a much more informed way than if the recorded
dependencies were just "A and B needs Collections") but if someone ELSE
has already tried this, and discovered A annd B works just FINE (who
knows, perhaps there is even a test suite to run!) then he/she can have
beaten the maintainers of A and B respectively and attached new
configurations to A and B that says that, according to him - A1.0 works
just fine with Collections1.1 (and the same for B).

So if you then trust this person, you can go ahead with even more
information to guide you - he said it worked, and since it was Dan
Ingalls - you decide to trust him.

And my third final point - if there was no Ingalls-config available, and
you did have to "take a chance" - then YOU can be the good citizen
paving the way for others to follow by verifying it works and attach a
new configuration to A and B.

Now - IMHO this is a GREAT model :). I am not saying we will not
discover details to tweak with it - for example, I can surely see it
extended with information about "intended dependencies" - meaning that
the maintainer might want to tell people what packages he wants to
depend on, this may not always be the same thing.

> I might be the main developer of package C, which also uses Collections.
>  Okay, so eventually tthe author of A, being a great citizen, upgrades
> their Collection package and reruns their tests -- shock, nothing brock.
>  While they are at it, they add a few class comments, and then post A
> 1.1 which depends on Collections 1.1.  Drats, I still cannot update my
> Collections library, because that will make me uninstall B.

No big difference, at this point - select Collections 1.1 and press
"install" it could say something (quite similar) like "The installed
package release B1.0 is only known to work with Collections1.0. A1.0 on
the other hand can be upgraded to A1.1 which is known to work with
Collections.1.1. Since Collections1.1 is marked by the maintainer as
being 'Code changes, but only bug fixes' B1.0 could still likely work,
do you still want to proceed and install Collections1.1 and also A1.1?"

Of course, exactly how the engine will talk to the user is a UI issue.
:) And upgrading A1.1 would of course be optional - the engine should
only be HELPFUL - not forceful. You should always be TOTALLY FREE to
install what the heck you like.

>  Note that I
> still cannot upgrade my Collections library, unless I uninstall B.

I can't really see where you have gotten this... "stiff bureacratic"
view of the planned dependency mechanism? :) Sure, you can upgrade
Collections at this point.

> Finally, suppose Collections gets updated to 1.2, and finally B gets
> around to doing an upgrade.  B 1.1 depends on Collections 1.2.  Now
> what?  I *still* cannot upgade B, because it depends on Collections 1.2,
> which is inconsistent with all available versions of A.

Ok, let's see where we are now:

Collections1.0
Collections1.1
Collections1.2
A1.0 -> Collections1.0
A1.1 -> Collections1.1
B1.0 -> Collections1.0
B1.1 -> Collections1.2

And you have Collections1.0, A1.0, B1.0 installed (the starting point
ignoring the potential upgrades I have described above). Now, we have a
few different ways to proceed. I will not describe all possible
variations - but let's try going to Collections1.2.

At this point - select Collections 1.2 and press "install" it could say
something like "The installed package releases A1.0 and B1.0 are not
known to work with Collections1.2. B1.0 can be upgraded to B1.1 which is
known to work with Collections.1.2. A1.0 can be upgraded to A1.1 which
is known to work with the previous release of Collections -
Collections1.1. Collections1.2 is marked by the maintainer as being
'Code changes, may break compatibility' A1.1 could still work, do you
still want to proceed and install Collections1.2 and then also A1.1 and
B1.1?"

>  I end up using
> A 1.0, B 1.0, and Collections 1.0.  And I still cannot update the
> version of Collections that my C package works with, because I can't
> install the Collections update anyway.  If I'm lucky, package A will
> release a version that works with 1.2 and I can upgrade everything.

As I have clearly shown this is not the case. I have NEVER argued for
the dependency engine to be some kind of POLICE. You can still install
*whatever you like*. The whole point is that you can do this and be
AWARE of what it means and what the risks are.

And I have also, I hope, clearly shown that since we can all attach
tested working configurations you will not be stuck at the whim of the
maintainers. And I also hope i have shown that the configurations can be
added AFTER the release, and even MODIFIED after the release. This is a
GOOD THING - because if they are wrong they should simply be fixed.

>  But
> notice: int hat case, every single package in the example has had to
> *synchronize* their releases, in order to reach a state where the jigsaw
> has a solution.  I could as well be unlucky, and A might declare itself
> compatible with Collections 1.3.....

Not commenting here since I hope I have shown that the "jigsaw" doesn't
NEED to be solved in full. If you have a solved jigsaw - then fine. If
you don't then fine too, at least you are aware of the different
dependencies in your image that haven't been tested by others.

> It appears that all the packages need to be upgraded in lockstep.  I
> can't offer you a proof for the more general case :), but I imagine it
> will only get *worse* as more packages enter the picture.  Is it

I don't think so. Of course, it all gets "worse" in some kind of way
with lots of intertwined packages - but I don't think the model will
suffer. There are some really nice things here helping it:

1. Users can help keep the dependencies tracked and verified, not only
maintainers. This is cool.
2. Pressure can build on key packages that really needs to be upgraded
in order to work properly.

This leads me to something I haven't talked about before - it should be
possible to record an "anti configuration" - meaning that "No, sorry, I
tested A1.1 with Collections1.2 and nope, didn't work". This would be
invaluable to have. Again, we record information - that is all. Then how
the information is used is purely up to you. I can even imagine
different engines or at least clearly different pluggable strategies. :)

> acceptible that every package needs to be tested with each incremental
> version of every other package, and thus achieve the lockstep
> progression?  That seems very inefficient if most package updates are
> compatibility-preserving bug fixes.

I have described that the compatibility level categorization of releases
will make this information explicit. If you don't want to test your
releases at all - then fine, others will do it for you. :) And if you do
test them, then why not record which releases you tested it against?
 
> Finally, notice that users almost always want the new versions of all
> packages, regardless of what the upsteam author has actually tested
> with.  Consider two cases:

I don't agree - but let me see what you are saying.

> 	1. They are drawing from a stable stream, e.g. 3.4.  All package
> updates are clear bugfixes, and further, all the testing that has
> happened has happened with the full set of packages loaded.

Eh... now you are talking about the update stream. That is only in part
related I think. 

> 	2. They are drawing from an unstable stream, e.g. 3.3.  Nothing works
> > anyway, even if it has been "tested" by upstream, so you may as well
> grab the newest stuff and hope for the best.  Back off on occasion if
> something breaks.

I don't think these are the only "cases" at all. For example, I may be
using 3.6 Basic with stable releases of KomHttpServer and HttpView (hah!
yeah, right) for my deployed app - but also using the latest and hottest
Monticello/Shout and Whisker because those are tools - and I want the
latest and greatest of those.

In short - I think one should be able to mix and still know as much as
possible about the risks and what has been tested or not.

Not sure what the cases were supposed to show either, that people always
want the latest? Not convinced at all.

> In short: dependencies on specific versions don't appear to help with
> their stated goal of reliability, they complicate the tools to an
> unclear degree, they appear to force development to be lockstep across
> all packages, and they actually *remove* the desirable functionality of
> upgrading all packages to their newest version.

Hehe, well - I sure hope that I have shown to at least some of the other
readers of this thread that the above description of my plan is simply
FUD.

To me, IMHO, they sure *does* appear to help, don't at all need to
complicate the tools (that is of course a tool issue), don't at ALL
force development to be lockstep, and doesn't remove anything at all.

> I don't think this is a good bargain.  There are already proven ways to
> get a reliable set of packages together, so let's use one of those
> instead.

IMHO I would say that SM has already proven itself in many, many ways.
It works pretty fine, even though it has limitations.

I want to give this a shot - and I invite everyone to help me in doing
this. And if we pull it off we will IMHO have a GREAT system that blows
much everything else I have seen out of the water.

And it will also be very suitable for test driven development. Did you
hear that Stephane? :) Because if we can attach or in some other way
associate how to run the tests with the packages (included or not) then
they will form a formal basis of testing out new configurations of
releases.

Hell, it could even be automated in theory - we could have a robot
server sitting trying to test new combinations of package releases and
attaching those autotested configurations automatically.

> > Why would I want to throw that information away? 
> 
> Sure, keeping the information is fine, and there are probably exciting
> things you can do with it.  Just don't use it as dependencies.

Well, I intend to use it as the recorded information about what releases
a specific release needs in order to function properly. :) 

--------------------------
continuing this reply:

> In the multiple maps question, I can boil the issues down to two
> scenarios.
> 
> Scenario 1: Someone wants to provide a mixin server that can be used to
> provide packages to multiple package universes.  How do you support
> this, while allowing end users to use several different mixin servers in
> the same image?

"provide packages to multiple package universes"? I am sorry, but I
don't understand exactly what that means.

Besides, the idea is for SM to be THE catalog for Squeak. It is NOT
meant to just be a copy of how things work in Debian. SM is a catalog
(sure, it has upload facility, but that is just an "addon") and NOT a
package repository.

In SM you know how many packages there are exactly. And the map is
coherent. In Debian you have a gazillion of unofficial repositories of
package files, you need to find them, list them in your sources.list and
then you still don't really trust the packages to work properly together
- because they are after all picked from here and there. And these files
don't share things - as the objects in SM does (SMAccounts, SMCategories
etc). I have asked you repeatedly how you would handle that.

I will support the tree structure I talked about - but it will still be
a coherent map, though with different visibility levels - but I am not
sure if that satisfies your need here.

And I have no idea why "while allowing end users to use several
different mixin servers" is such an important thing here. Yes, I know
you like Debian and sources.list - but what is so cool about it really?
>From a user standpoint it sucks. Why should I need to find and list
several sources when instead there could just be a full complete catalog
that I can look in?

Decentralization is good if it fulfills a goal. In Squeak I would say
the centralization of SM is a GOOD THING. We have a SINGLE map. And the
multiple-servers-in-a-tree will fulfill important goals - I don't intend
to add that just to make things more complex. But it will still maintain
the sense that there is ONE map, although with the ability to have
privately local added stuff.

> Scenario 2: Your local organization makes its own SqueakMap server, with
> potentially its own user accounts, packages, and policies about package
> updates.  The local server wants to be an extension of Squeak 3.7final. 
> Can the organization do this without *any* interaction with a central
> Squeak authority?

When you say "makes its own" - do you mean set up a node in the tree I
was talking about, or do you mean "implement their own code"? Just
curious because I am not sure what you mean with "policies about package
updates".

And again - the servers in the tree are NOT partitions based on Squeak
version.

But if you give me some slack in interpretation here - yes, a company
will be able to set up a SM server on its own and hook it into the tree
by telling it which parent server it has (typically the master). Then
that server can have SMObject instances which are only visible to
servers below it (that includes the clients).

So the answer is yes, but do note that these accounts and packages will
NOT be visible to others, it will be purely for the local organization.
So if the company want things to be visible to the world - then why not
register those objects in the master catalog? It is a *catalog* - not a
repository.
 
> Both 1 and 2 are trivial if the client pulls from a list of maps instead
> of just one map.

No it is NOT trivial. Either we give up a bunch of stuff we have today -
and make SM just be a copy-of-Debian, and sure - *then* it may be easy
(trivial is a strong word). But if we want to keep a lot of the things
we have today then it is DEFINITELY NOT trivial.

> Scenario 2b: What if HP and Compaq each have a local SqueakMap server,
> and then HP acquires Compaq and they want to merge the server contents
> together?  There is some inevitable pain from this, but how can we
> minimize it?  Multiple maps help on 2b, though there are more issues to
> consider as well.

Multiple maps help? First of all - this scenario is very uninteresting.
I can't see why we should plan for that. Secondly - since SMObjects most
likely will be movable "up" the tree (IMHO a very likely scenario - a
private package going public) it should be very easy to put in a third
server as parent of both - and then move the objects there.

But again, not very interesting to discuss.

> These said, here are some individual comments:
> 
> > > With multiple maps, you immediately gain the following
> > > abilities:
> > > 
> > > 	1. Keeping track of which package releases are in which Squeak
> > > versions, with a UI that requires no extra activities by the users.
> > 
> > I can't understand what you mean. With multiple maps you will have to
> > choose the "3.6" map in order to see the "3.6" packages, right? And with
> > multiple maps I will need to select which maps to register a release in,
> > depending on which Squeak versions it works in, right? Again, no
> > difference compared to what we have now.
> 
> You have the general idea, though are missing one important difference. 
> With multiple maps, each image can *implicitly* specify a universe it is
> operating in via the list of maps it is pulling from.  Then all the
> tools need no further interaction with the user.  Both when downloading
> packages and when uploading packages and when uploading thumbs-up
> notifications, the tools will Just Know what to do.

Again, this is just a tools issue. You can already tell the package
loader to only show you packages/releases for say 3.6. There is no
"further interaction".

I simply can't understand why it is beneficial to split the map (it is
still a catalog - not a repository which you over and over seem to
forget) over multiple servers and also doing so using a single
partitioning - namely which Squeak version a package is for.

What about splitting it based on how stable the releases are? What about
splitting it based on ANY other attribute? And then - realizing this -
why splitting it at all? What is the gain?

The packages (all SMObjects, not just packages) are already
categorizable using the category tree. We can slice them up in a whole
range of different ways TODAY.

The mere physical location of the bits on the Internet is hardly
interesting, is it? (again, remember that SM is a catalog, not a package
repository).

> Anyway, even if you disagree that the multiple maps approach is a better
> result, the really big point is that multiple maps gives it to you with
> no further code.

I do disagree, and I don't see what it gives me at all. :) I can already
pick stuff based on Squeak version and a number of other criteria.

> > > 	2. The ability for users to set up their own package universes without
> > > needing to coordinate withany central authority.
> > 
> > This is planned. The idea is to have a tree of map servers in which you
> > add local additional information, thus you can have local packages only
> > visible to you, or your company etc.
> >
> > But this has NOTHING to do with which Squeak version things are meant
> > for. It is purely a visibility thing - and the servers are still clearly
> > connected with each other in a tree. 
> 
> I don't understand your "nothing" comment, because visibility and
> versions-meant-for (I'd prefer universes-meant-for) are the same thing. 

Eh, no it is definitely not the same thing. Maybe I wasn't clear.
Visibility is the fact that you even know that there is a package named
Foo. It is visible to you. It is thus not a purely private package only
visible inside Bluefish. What Squeak versions it works in is a totally
different thing.

In short - I am talking about public or private packages (and other
SMObjects).

> But I'll procede on the parts I understand.
> 
> Trees of servers are very close to what I am talking about,  but they do
> not give you the full functionality that multiple maps do.  In
> particular, they do not support "mixin" servers such as the ones in
> Debian for DVD software and updated Java plugins, etc.  There are
> hundreds of these mixins nowadays, so it is a valuable feature to
> support.  To support mixin servers, though, you must allow the nodes of
> the trees to have multiple parent nodes.  And once you do that, you are
> doing exactly what I suggest, only at the server level instead of in the
> clients.

I need much more information on what such a mixin server is supposed to
give us before commenting further. I emplore you to remember that SM is
a CATALOG. Let me say that once more - A CATALOG. Like dmoz for example.
Are there compelling reasons for us to split up a catalog over multiple
machines? No, the only reason for doing that is to handle private
information IMHO.

But to repeat - the server tree is purely meant for visibility. That is
after all IMHO the only interesting "attribute" of the SMObjects to
consider. If an SMObject isn't meant to be seen or even known about in
public, there simply is no point in putting it on the master server. But
if it is meant to be seen and known then it should be there.

Note also that the clients will use the same code as the servers - just
as today. They are "local servers" so to speak.

> > 2. This is hard stuff to get working properly, unless we are going to
> > throw away stuff we have today.
> 
> I agree that difficulty of implementation is a important.  However, it
> needs to be compared with the difficulty of implementing alternatives. 
> Also, isn't it bothersome if a union operation is difficult to
> implement?

Eh, care to elaborate?

I am just saying that the current SM model, which is a rather complex
graph of objects including SMAccount and SMCategory (with more to come),
is NOT "trivially" split up over multiple machines. That ought to be
pretty obvious to anyone. And when I even can't see the point in doing
the split (except for the planned tree structure to handle private
stuff) then why would I even contemplate doing it?

> > > 	4. The ability for a package to be maintained by different people in
> > > different package universes, e.g. if I only support Chuck in 3.7, Joe
> > > can volunteer to support it in 3.0.
> > 
> > This already works today with co-maintainers.
> 
> It mostly works, though it currently requires central coordination. 
> What if it's Jose wanting to support the Spanish version of Chuck?  Jose
> should not need permission from me, and he should be allowed to call the
> package "Chuck" on the Spanish SqueakMap.  More on this below.

Well, he can do it in different ways today (well, #3 in the future):

1. Take your "Chuck", and create his own package named "Spanish Chuck"
and go for it. That is a separate package with a unique name. No
"coordination" needed.

2. Join as a co-maintainer and make spanish releases of Chuck. Would
need coordination in that you need to allow him as a co-maintainer - but
that seems pretty reasonable to me.

3. Could in the future make unofficial releases of Chuck without your
consent. They would be somehow linked from Chuck so that people can find
and install them, but they would be owned by Jose.

Now, I don't want to allow non-unique package names. It will just
confuse things.

If someone wants to set up a spanish SqueakMap with spanish packages
using spanish descriptions - then I personally would think that the best
thing is not to do that and instead make SM handle it. Let us then add
categories for that. I would still be very interested in seeing those
packages, and there are many people multilingual.

Just look at dmoz - it is a perfect example. It has lots of Swedish
sites in it - and I can find them by looking in the correct category.
Why would lots of small local dmozes gain except confusion and me not
knowing if I have looked in all the right places?

> > > I find this list striking, when one considers that all of these features
> > > happen automatically and with no coding at all.
> > 
> > That is not true.
> 
> I meant, no coding beyond the coding necessary to get multiple maps.  Do
> you disagree that multiple maps would give you the things I describe,
> with no further coding?

I meant that the coding necessary to handle a distributed SM model over
distinct separate servers that don't know about each other and still
maintain non-redundancy and handle the complex object graph that SM has
today would include a LOT of quite complex coding.

And then, to gain what? AFAICT we gained nothing. I can't possibly see
how the mere distribution of bits onto multiple servers give us
benefits. It only gives me headache.

It itches in my fingers again.... ok, can't help it: SM is a CATALOG.

> > > Now, perhaps I am misunderstanding the purpose of SqueakMap.   If
> > > SqueakMap wants to be a catalog of everything, then I have misunderstood
> > > and that is fine.   In that case, however, I do suggest that we start
> > 
> > Have you read the article I recently posted on SqP? It explains quite a
> > bit about what I want SM to be. And yes, it was just recently posted so
> > you may have not seen it yet.
> 
> No, I haven't.  Thanks for the pointer; I look forward to reading it.
>  
> > > work on a new tool which is a "package universe browser".  Users should
> > > have some tool that lets them select packages from a menu, install them,
> > > and have it just work.
> > 
> > Hmmm, and what is the package loader you think?
> 
> It could be either one.  Do you see what I mean between the difference
> of a "catalogue of everything" versus the toolset that would support
> package universes?  SqueakMap clearly started life as the former, and it
> sounds like it is now trying to be both the former and the latter.  It
> may simplify things to divide SqueakMap into two parts, one for each of
> these purposes.

I humbly disagree and instead think that SM, just like dmoz, should
instead be extended to handle the various attributes we want to put on
the SMObjects so that it can sliced, diced and peeled just the way we
want to. I can't see any good reason (except for the visibility thing
mentioned in the next paragraph) to split the SM catalog over multiple
machines.

So the ONLY reason for adding the tree-structure I am talking about is
to enable the ability to have "added local private SMObjects".

> > > Regarding versioned dependencies, Julian, Goran did indeed call the
> > > configurations "dependencies".  And please consider that in the A,B,C
> > 
> > I don't know when I did - it sounds "wrong" but I may have slipped my
> > tongue sometime. Can you point me to it?
> 
> I am glad that you are not calling these dependencies any longer.

Calling what? I am totally lost. I have called them "configurations" for
quite some time.
A configuration is simply a tested "configuration" of a specific release
in that it lists the dependencies in the form of the specific releases
it works with.

> However, when I made my post earlier, you did halfway object to my
> request to have unversioned dependencies:
> 
> 	http://lists.squeakfoundation.org/pipermail/squeak-dev/2004-July/080026.html

I am not sure what you are referring to. In that posting you asked for
dependencies to be expressed between packages instead of package
releases, and I said that I will take it into consideration. What I
meant is that there may be some kind of relation that you want to
express in that way - that was the thing I was blabbering about in
"intended dependencies".

For example, "SharedStreams" has no intended dependencies at all - which
means it should work fine in a vanilla image. If I mistakenly had made
it dependent on some other package, then it would be nice to know this
fact that no - there should not be any such dependencies.

But I am still robustly unshaken in my belief in the model of
configurations that I have explained in painful detail in this posting.
:)

> Also, here is a direct quote from a while back where you call it a
> "dependency conflict" that the configurations have not been solved
> happily:
> 
> "Hey, Mungo 1.3 has a dependency conflict with Pingu 1.2. They both rely
> on Zingo but Pingu has only been verified to work with Zingo 2.0.
> According to the compatibility level of Zingo 2.0 it is categorized as
> *compatible* (=level 4, changes have been made but the maintainer says
> he hasn't changed the API so it should work), would you like to use that
> instead?"
> 
> If they aren't dependencies then why do they give a "dependency
> conflict"?

Hmmm? Ok, we are clearly not communicating. I said above that I don't
think I have used the word "dependency" when I was in fact describing
the concept I call "configuration".

An attached set of configurations to a specific release describes the
dependencies it has - or rather - the set describes the tested known
working "environments" for that release and thus I would say it does
express the "dependencies" of that release.
 
> > The configuration model I have in mind is more about having users and
> > maintainers "recording" working configurations so that other people,
> > using an engine, can install packages and needed dependencies in an as
> > easy way as possible. 
> 
> Here you are calling them dependencies again....

No, here I was referring to the *actual package releases* needed to
install in order to satisfy the needs of a set of selected
packages/releases. Sure, perhaps sloppy wording. Perhaps I should have
written:

"...so that other people, using an engine, can install package releases
and the different package releases they need in order to work in an as
easy way as possible."

Whatever.

> > 	- It must be very easy to both:
> > 		- Record a working config
> >
> > 		- Deviate from the information so far collected.
> 		
> Sure.  Please remove the "must", however.  And by the way, it is

Why remove "must"? I really feel it MUST be easy. If it is hard, then it
may very well fail.

> interesting that you consider the second so important, when Debian is
> doing quite well without it.

Well, I don't consider Debian to be the super-duper solution - sure,
it's good - but it is not perfect. Most people are aware that you can
quite easily end up in problems with Debian too, especially if you try
to mix things from testing into stable etc.
 
> > > example there was *never* a time that I could upgrade *any* package.  If
> > 
> > This sounds very wrong. Julian responded on this topic and it is
> > probably purely useless for me to reiterate what he wrote.
> 
> What would be useful would be for someone to make a concrete proposol of
> what should happen in examples like the one I described.  No one has
> done that.  They simply say the configurations should be ignored. 
> Doesn't it beat the purpose, though, if all this great information is
> simply ignored by the tools?

I hope I have explained it in a good way now.
 
> > I have tried to explain these ideas many times. I think it will work.
> 
> And I hope you succeed.  This is a wonderful rallying center of our
> community.

Yes, it is. And I also want to reiterate that there are two major
reasons IMHO that SM has worked so far:

1. It is a catalog, not a repository. It is format agnostic, and even if
it may "slip" here and there it is truly meant to be the Good
Cooperative Guy in the community.

2. #1 has led to the community accepting SM as the singular catalog for
Squeak. This is important.  A catalog like this is only Truly Really
Good if it is a singularity.

> > Debian is nice, but it ain't perfect.
> 
> It is also proven to work, as I'm sure you agree.

As is SM, as I am sure *you* agree. :)
 
> > > decentralized playground, though, and everyone is free to play in it
> > > however they like.
> > 
> > Of course. But SM is such a central piece to the community, that I want
> > as many as possible to like the way I am pushing it. I can't of course
> > convince everyone - but just so you know - this is the main reason I am
> > discussing this so much on this list.
> 
> One last thing.  After reading this post and skimming your Squeak People
> article, I see that you are trying to centralize various aspects of
> SqueakMap.  This seems like a good idea, but let us please be wary of
> building centralist policies into the architecture itself.  Squeak is a
> loose community to begin with, and it seems overly ambitious to try and
> force people to organize more rigourously when they haven't decided to
> already.

I am listening, it is a delicate balance of course.
 
> Let me give some specific examples.  Here are some things that should be
> doable at a local level, without involvement from any central authority:
> 
> 	1. The creation of new universes that draw packages from an existing
> one.  The local guys should not have to coordinate with the server you
> are drawing from.

If I understand you correctly, that is planned with the tree-of-servers.
You will for example be able to have 54 packages only visible inside
Bluefish but they are in every other way "connected" to the global map.
This is the part that is tricky of course.
 	
> 	2. The creation and maintenance of user accounts.  Individual
> organizations should be able to have their own accounts without
> publishing them publically.

That is planned with the tree-of-servers.

> 	3. The designation of who has permission to post package versions to
> each server.  Particular package universes should have their own rules
> about this, that even the original owner of a package cannot necessarily
> override.  (Even though in the main Squeak repository, we want to give
> more deference to the designated package owner.)
> 
> You may not be surprised at this point, but multiple maps gives you
> these decentralized policies automatically, and also, Debian manages its
> policies in a way that gives the above properties.  While these
> properties are not strictly necessary, they are both desirable and
> attainable.

You know, if we consider the attribute "server of package X" - meaning
which server it is placed on (you keep talking in terms of repositories
but whatever) then we can EASILY in SM replace this attribute with a
categorization.

Let's say I log into SM today and add the top category "Package
universe" and then add... well, I am not sure - what partitioning we
want that we don't have already... well, just to get my point through, I
call them:

"Server 1"
"Server 2"
"Server 3"

Ok, now we can "put" the packages in one (or more) of these "universes"
simply by adding any of these categories.

So, what did we gain? We already have a range of categorizations. Again,
I can NOT understand why the mere fact that something is strewn around
on separate machines gives us an advantage. It is just painful in terms
of implementation

And I still note that you haven't explained how these multiple servers
should synchronize the shared parts of the object model. I have asked
repeatedly how SMAccount, SMCategory etc will work in a coordinated
synchronized fashion. SM is a more and more complex object model -
distributing it with read/write capabilities is NOT trivial, and I have
repeatedly said so.
 
> -lex

regards, Göran

PS. This posting took a considerable time to write, unless I find
something really interesting in the coming replies I will not take the
time to reply in such detail. I will reply though.



More information about the Squeak-dev mailing list