Hi Lex,
I have a question regarding package universes.
Assumed there are separated universes for stable and unstable packages, U_stable and U_unstable.
Now we put the union of these packages into a new universe U_all, but to know, where they have come from, we label each package with the name of the universe, from which it stems. As a result each package in U_all has a label #U_stable or #U_unstable.
Now to my question:
I see no difference between the functionality of - a dependency resolver working in U_stable (or U_unstable), and - another one working in U_all *together* with a filter selecting the wished packages with label #U_stable (or #U_unstable), before starting to work.
Do you see any problems with this approach?
Greetings Stephan
Stephan Rudlof sr@evolgo.de wrote:
Assumed there are separated universes for stable and unstable packages, U_stable and U_unstable.
Now we put the union of these packages into a new universe U_all, but to know, where they have come from, we label each package with the name of the universe, from which it stems. As a result each package in U_all has a label #U_stable or #U_unstable.
Now to my question:
I see no difference between the functionality of
- a dependency resolver working in U_stable (or U_unstable), and
- another one working in U_all *together* with a filter selecting the
wished packages with label #U_stable (or #U_unstable), before starting to work.
Do you see any problems with this approach?
Yes, I agree that you could get the dependency resolution to work the same here. You can indeed implement a universe as a filter over a catalog of everything, if you are careful.
It does lead to some complications, though. Suppose the "TweakBase" package wants to be called "Base" in the "Tweak" universe? To make the name-shortening work, you need to either let packages have different names under different filters, or you need to make two separate entries for "TweakBase" and Base". The former case is complicated, and the latter case seems to defeat whatever purpose there is of making universes be filters.
It also does not give you the decentralization properties that the 2-day universes prototype gives you. It does not, for example, allow Disney to have a private server for Fantasia 2007 stuff.
By the way, here's a little thing to think about from the Debian world: few Debian package use the original upstream package bit-for-bit identically. It frequently, at the least, installs files into different locations than the upstream "make install" would do. If this happens in Squeak, and if we implement universes as filters over an underlying catalog of everything, then the catalog of everything will fill up with thing like "Chuck for 3.7", "Chuck for Tweak", "Chuck for 3.6", etc., which are all minor variations of each other. That doesn't seem like an improvement over handling all the "for foo" variants in separate catalogs.
-Lex
Lex,
lex@cc.gatech.edu wrote:
Stephan Rudlof sr@evolgo.de wrote:
Assumed there are separated universes for stable and unstable packages, U_stable and U_unstable.
Now we put the union of these packages into a new universe U_all, but to know, where they have come from, we label each package with the name of the universe, from which it stems. As a result each package in U_all has a label #U_stable or #U_unstable.
Now to my question:
I see no difference between the functionality of
- a dependency resolver working in U_stable (or U_unstable), and
- another one working in U_all *together* with a filter selecting the
wished packages with label #U_stable (or #U_unstable), before starting to work.
Do you see any problems with this approach?
Yes, I agree that you could get the dependency resolution to work the same here. You can indeed implement a universe as a filter over a catalog of everything, if you are careful.
It does lead to some complications, though. Suppose the "TweakBase" package wants to be called "Base" in the "Tweak" universe? To make the name-shortening work, you need to either let packages have different names under different filters, or you need to make two separate entries for "TweakBase" and Base". The former case is complicated, and the latter case seems to defeat whatever purpose there is of making universes be filters.
In spite of this I don't see a problem here. Packages exist in different releases/versions.
Normally, I think, one release of "Base" exists in just one universe: eg. "Base_4.1.2.3" in just universe U_1; having a label #U_1 for the filter.
Now to the more complicated case of a package release (one version of a package) belonging to *two* universes: then you just have two labels, say #(#U_1 #U_2) to flag this release as belonging two two universes. The filters have to be written to deal with collections of labels, of course.
In both cases the base name of the package could be "Base" without any problems.
What am I missing here?
It also does not give you the decentralization properties that the 2-day universes prototype gives you. It does not, for example, allow Disney to have a private server for Fantasia 2007 stuff.
Why not?
Assumed there is one main server and two local ones:
main
local1 local2
. The main one is parent of both local1 and local2. Then you just need a possibility to build a union of local packages belonging to local1 and local2 and the packages of main, and afterwards you are running the filter. This could be accomplished by downloading the package descriptions of main to the locals and adding the local package descriptions to them (e.g.) before filtering according the attributes.
Where is the problem?
By the way, here's a little thing to think about from the Debian world: few Debian package use the original upstream package bit-for-bit identically. It frequently, at the least, installs files into different locations than the upstream "make install" would do. If this happens in Squeak, and if we implement universes as filters over an underlying catalog of everything, then the catalog of everything will fill up with thing like "Chuck for 3.7", "Chuck for Tweak", "Chuck for 3.6", etc., which are all minor variations of each other. That doesn't seem like an improvement over handling all the "for foo" variants in separate catalogs.
I see it so: the package *releases/versions* have attributes saying in which universe they make sense. One attribute per universe.
If you want to be more finegranular regarding installing locations: what does this mean regarding Squeak? There are less files... Theoretically you could also replicate install methods/scripts/whatever, one for each universe, if this would make sense (again filtered by the universe attribute). But this would be the same replication as you would have for different universes (each universe its special script).
Again: where is the problem?
Greetings Stephan
-Lex
Stephan Rudlof sr@evolgo.de wrote:
lex@cc.gatech.edu wrote:
Yes, I agree that you could get the dependency resolution to work the same here. You can indeed implement a universe as a filter over a catalog of everything, if you are careful.
It does lead to some complications, though.
[...]
In spite of this I don't see a problem here. Packages exist in different releases/versions.
Normally, I think, one release of "Base" exists in just one universe: eg. "Base_4.1.2.3" in just universe U_1; having a label #U_1 for the filter.
Now to the more complicated case of a package release (one version of a package) belonging to *two* universes: then you just have two labels, say #(#U_1 #U_2) to flag this release as belonging two two universes. The filters have to be written to deal with collections of labels, of course.
I agree that this could work, once some details are fleshed out. I am simply saying it is more complicated.
It also does not give you the decentralization properties that the 2-day universes prototype gives you. It does not, for example, allow Disney to have a private server for Fantasia 2007 stuff.
Why not?
Assumed there is one main server and two local ones:
main
local1 local2 [etc.]
Yes, but this is an extra feature beyond what you described above. Note my wording carefuly: it does not *give* you the decentralization properties in itself.
If you are willing to have multiple servers, then I see no advantage to the design you described with #U_1 labels, etc. Once you allow multiple servers, you can set things up like this:
main-3.4-stable main-3.6-stable main-devel tweak-devel
Now there's no need for the stuff you describe about packages having multiple names, descriptions, dependencies, etc. You can use simple package descriptions. If different universes want a minor change to the package description, then you simply post a slightly different package descriptions to each server.
-Lex
lex@cc.gatech.edu wrote:
Stephan Rudlof sr@evolgo.de wrote:
lex@cc.gatech.edu wrote:
Yes, I agree that you could get the dependency resolution to work the same here. You can indeed implement a universe as a filter over a catalog of everything, if you are careful.
It does lead to some complications, though.
[...]
In spite of this I don't see a problem here. Packages exist in different releases/versions.
Normally, I think, one release of "Base" exists in just one universe: eg. "Base_4.1.2.3" in just universe U_1; having a label #U_1 for the filter.
Now to the more complicated case of a package release (one version of a package) belonging to *two* universes: then you just have two labels, say #(#U_1 #U_2) to flag this release as belonging two two universes. The filters have to be written to deal with collections of labels, of course.
I agree that this could work, once some details are fleshed out. I am simply saying it is more complicated.
It is just introducing 'universes' at one map. Dealing with multiple maps just to separate universes gives other problems (Göran has written about this topic).
It also does not give you the decentralization properties that the 2-day universes prototype gives you. It does not, for example, allow Disney to have a private server for Fantasia 2007 stuff.
Why not?
Assumed there is one main server and two local ones:
main
local1 local2 [etc.]
Yes, but this is an extra feature beyond what you described above. Note my wording carefuly: it does not *give* you the decentralization properties in itself.
I've focused at the privay point here: if I've correctly understood Göran, he is planning to take this into account by the planned tree structure.
If you are willing to have multiple servers, then I see no advantage to the design you described with #U_1 labels, etc. Once you allow multiple servers, you can set things up like this:
main-3.4-stable main-3.6-stable main-devel tweak-devel
You always can do this in one map: but you know this.
Now there's no need for the stuff you describe about packages having multiple names, descriptions, dependencies, etc. You can use simple package descriptions. If different universes want a minor change to the package description, then you simply post a slightly different package descriptions to each server.
To 'package descriptions'. I had written (to the Disney private server point, see above):
* Why not? * * Assumed there is one main server and two local ones: * * main * * local1 local2 * * . The main one is parent of both local1 and local2. * Then you just need a possibility to build a union of local packages * belonging to local1 and local2 and the packages of main, and afterwards * you are running the filter. This could be accomplished by downloading * the package descriptions of main to the locals and adding the local * package descriptions to them (e.g.) before filtering according the * attributes. * * Where is the problem?
Here 'adding the local package descriptions' means adding information about local packages (all information needed for installation!) residing at a local server.
Greetings Stephan
-Lex
Hi all!
lex@cc.gatech.edu wrote:
Stephan Rudlof sr@evolgo.de wrote:
Assumed there are separated universes for stable and unstable packages, U_stable and U_unstable.
Now we put the union of these packages into a new universe U_all, but to know, where they have come from, we label each package with the name of the universe, from which it stems. As a result each package in U_all has a label #U_stable or #U_unstable.
Now to my question:
I see no difference between the functionality of
- a dependency resolver working in U_stable (or U_unstable), and
- another one working in U_all *together* with a filter selecting the
wished packages with label #U_stable (or #U_unstable), before starting to work.
Do you see any problems with this approach?
Yes, I agree that you could get the dependency resolution to work the same here. You can indeed implement a universe as a filter over a catalog of everything, if you are careful.
It does lead to some complications, though. Suppose the "TweakBase" package wants to be called "Base" in the "Tweak" universe? To make the name-shortening work, you need to either let packages have different names under different filters, or you need to make two separate entries for "TweakBase" and Base". The former case is complicated, and the latter case seems to defeat whatever purpose there is of making universes be filters.
The above problem doesn't really sound like a big issue to me. Squeak packages/projects tend to want to have unique names anyway - simply because the Squeakers, we, would otherwise get confused. Implicitly there is just one "namespace" for public Squeak packages.
For private "project maps" on the other hand I agree it might be useful to be able to use context dependent generic names like "base", but then... well. I do think it should be possible to have local servers with private content - I haven't denied that. I just don't believe in multiple public maps - especially not along these "axis" (stability etc) which just as easily can be modelled as attributes on the objects in a SINGLE model (single *model* ~= single *server*).
It also does not give you the decentralization properties that the 2-day universes prototype gives you. It does not, for example, allow Disney to have a private server for Fantasia 2007 stuff.
By the way, here's a little thing to think about from the Debian world: few Debian package use the original upstream package bit-for-bit identically. It frequently, at the least, installs files into different locations than the upstream "make install" would do. If this happens in Squeak, and if we implement universes as filters over an underlying catalog of everything, then the catalog of everything will fill up with thing like "Chuck for 3.7", "Chuck for Tweak", "Chuck for 3.6", etc., which are all minor variations of each other. That doesn't seem like an improvement over handling all the "for foo" variants in separate catalogs.
But if the (as you like to call it with a slight negative touch) "catalog of everything" is indeed a catalog of EVERYTHING, then isn't that exactly what you want? :)
Anyway, I don't consider this issue a real "issue" either. IMHO these variants of a package are simple different branches of releases of it.
Now to my final more urgent meta question:
I am worried about this discussion. I briefly looked at the universe code and noted that it in many ways it duplicates concepts etc from SM. Now, I worry about "forking" in these fundamental architecture projects. IMHO one of the greatest things with SM is that we have managed to stand behind it in a united fashion and that it thus has become THE place to look for stuff.
As I have said before - the whole idea with a catalog is diminished if we have 10 different catalogs all scattered out and using different tools. I really, really do think we should strive for focusing on ONE. That is also why I am so intent on *collaboration* in developing SM.
This doesn't mean SM should not support private "Disney" catalogs etc, what I mean is that:
- I think it would be better if we focused development efforts on SM. - I think it would be better if there is a user experience of a single catalog that people use.
So given the discussion I will contemplate how to be able to have multiple "source" catalogs mixed and merged into the client including the update/commitability I am striving for. It will very likely end up with a whole range of questions about limitations - but I will try it, without loosing the "sense" of SM being a singularity.
One thing for example is that if we DO make SM capable of this then surely we would like SM to be able to keep track of all these other "sub maps" so that I as a user isn't condemned into finding them "manually" using Google. That would just put us right back where we were before SM.
So... in short: Lex - if you want to work on these issues would you consider working with me on it in SM, or have you already decided to "roll your own" and try to "compete" with SM?
It would be very good for me to know, and it would also be very good for me to know what others think.
Also - I don't agree fully with Stephan on his model, but I still don't want him to go off and roll his own. It is better to be able to complement each other and work together. And to enable SM to be a base for it.
-Lex
regards, Göran
I don't mean to split things off. I actually proposed these ideas as future directions for SqueakMap, and only implemented them when you argued that they are bad ideas. Most any future direction is fine with me, at this point, so long as we please use simple working solutions until something better comes along.
Some possible directions I'd be very happy with:
1. SM could incorporates the universes model somehow. There is just one important idea that keeps this from going smoothly; see below. 2. You or someone else could pick up the Universes code; it doesn't need to be me at all, and in fact, I'd rather it were not me. 3. SM and Universes could have crosslinks between them so that it's easy to post to both of them simultaneously. This would be a handy addition to SqueakSource, for example.
In the meantime, I certainly suggest that people keep posting stuff on SqueakMap no matter how much they use Universes. They are complementary tools.
I also intend to create a 3.7 stable universe in the next few weeks, because this is something Universes would seem to work well at, and it's something that will give people a good idea of how Universes works.
For the development stream, the best strategy is harder to see. I'd guess that dependencies are a major thing to think about, but the conclusion you draw depends on how your and Stephan's dependencies come along. Let's leave aside that discussion for a future email, because there's something more important I'd like to talk about.
Please consider carefully this one issue: does *everything*, down to every last minor variation of every package anyony posts for any purpose whatsoever, really need to be in the same index of packages? As a mind experiment for what this map will be like, imagine from the Linux world that someone made a big index holding every package in every version of Slackware, Debian, Redhat, Gentoo, and OpenBSD. My mental picture of this has a lot of minor variations of most packages. There will be a dozen or more gcc-2.95's.
As another mind exercise, suppose we made a universe that includes the current frontier of SqueakMap's packages. This would not include every package, but it would include one or two version of quite a lot of them. In particular, it would include everything that one can currently reasonably expect to be installable into a 3.7 image. Every 3-12 months we could fork off a stable universe that only receives bug fixes and which has any horribly broken stuff removed from it; the unstable universe would continue on and would still have all of the same packages in it.
Doesn't the first scenario sound a little hard to manage? And doesn't the second one sound just fine? In the second scenario, users in both stable and unstable images would see precisely the set of packages that are reasonably maintained and installable. Hackers that additionaly want to see unmainained packages can still do it, though they must work a little harder and use some sort of "multiverse browser" or access a catalog of everything.
I ask about this one issue because if you let go of the idea of having absolutely everything in one index, then the rest of the universes approach seems to mesh very nicely with where Squeak Map seems to be going. It becomes straightforward to let maps be merged into larger maps, and to let non-central maps be locally administered with their own accounts and policies. We could then have simple dependencies, stable releases, mixin servers, private servers, and local update policies, all without requiring any further cleverness.
Finally, let me be very clear on one other thing: all of the efforts on things like dependencies, interfaces, compatible updates, branching versions, and simultaneous loading of multiple versions, are still very important no matter what we do in the short term regarding package catalogs. The present discussion is just about what the next step or two should be.
In summary, I would be happy to have one tool or toolset we all agree on using. But while the really great Grand Unified Tools are still being developed, shouldn't we use simple existing solutions and make do as well as possible?
Regards,
Lex Spoon
Hi Lex and all!
Note: Tomorrow we are leaving to China so my mind is "elsewhere" but I wanted to wrap up these issues so that I can go "offline" for the next 14 days and not worry. :)
lex@cc.gatech.edu wrote:
I don't mean to split things off. I actually proposed these ideas as future directions for SqueakMap, and only implemented them when you argued that they are bad ideas. Most any future direction is fine with me, at this point, so long as we please use simple working solutions until something better comes along.
Simplicity has always been something I strive for in SM. As you recall SM1 didn't even have releases - and that was "too simple". But it was a good way to get things started so I don't regret it.
Some possible directions I'd be very happy with:
- SM could incorporates the universes model somehow. There is just
one important idea that keeps this from going smoothly; see below.
Number 1 here would be the best IMHO. :) I promise to look long and hard at your proposal (I have already done so, but I want to be sure we *all* understand the pros and cons here) and I also promise to try to "meet the requirements" as long as I find an interest for them in the community.
For example - the idea of being able to set up private "submaps" (or whatever) or even public ones is a feature that I *want* to offer - I just *also* want to avoid all the negatives of such a model, see below.
Perhaps a good way to go forward here is to start talking about use cases? What are the use cases we want to support?
- You or someone else could pick up the Universes code; it doesn't
need to be me at all, and in fact, I'd rather it were not me.
- SM and Universes could have crosslinks between them so that it's
easy to post to both of them simultaneously. This would be a handy addition to SqueakSource, for example.
In the meantime, I certainly suggest that people keep posting stuff on SqueakMap no matter how much they use Universes. They are complementary tools.
Well, I would say they are competing tools. I may be wrong.
I also intend to create a 3.7 stable universe in the next few weeks, because this is something Universes would seem to work well at, and it's something that will give people a good idea of how Universes works.
Ok, just as long as I know what you intend to do. I interpret this as a kind of "fork" and well, I will just have to make sure SM can stand up to the challenge.
For the development stream, the best strategy is harder to see. I'd guess that dependencies are a major thing to think about, but the conclusion you draw depends on how your and Stephan's dependencies come along. Let's leave aside that discussion for a future email, because there's something more important I'd like to talk about.
Sure. In short - I have my plan clear in my head and partially working in code. Stephan's plan adds more mechanisms "on top" (as I see it) and thus, they seem worthwile to explore. If my planned simpler model will suffice or not - we will see.
Please consider carefully this one issue: does *everything*, down to every last minor variation of every package anyony posts for any purpose whatsoever, really need to be in the same index of packages? As a mind experiment for what this map will be like, imagine from the Linux world that someone made a big index holding every package in every version of Slackware, Debian, Redhat, Gentoo, and OpenBSD. My mental picture of this has a lot of minor variations of most packages. There will be a dozen or more gcc-2.95's.
The comparison isn't "fair" IMHO. I could argue the other way and say "Does all the 12000 packages (or how many they are now) in Debian really need to be in one repository?".
What is large and what is small? What are the factors that would define the "borders" of SM? Well, hard to say. Now, as a Squeaker I am not interested in a VA package (and likewise a Debianee isn't interested in a Redhat specific package). etc etc.
Btw - "same index of packages" above doesn't really say anything about the architecture of it. For example - DNS could be argued to be a single index - right? But of course technically it is highly distributed - but for the user it acts as a single logical entity.
As another mind exercise, suppose we made a universe that includes the current frontier of SqueakMap's packages. This would not include every package, but it would include one or two version of quite a lot of them. In particular, it would include everything that one can currently reasonably expect to be installable into a 3.7 image. Every 3-12 months we could fork off a stable universe that only receives bug fixes and which has any horribly broken stuff removed from it; the unstable universe would continue on and would still have all of the same packages in it.
Doesn't the first scenario sound a little hard to manage? And doesn't
Not really, it has worked great the last two years. :)
the second one sound just fine? In the second scenario, users in both stable and unstable images would see precisely the set of packages that are reasonably maintained and installable. Hackers that additionaly want to see unmainained packages can still do it, though they must work a little harder and use some sort of "multiverse browser" or access a catalog of everything.
This kind of reasoning doesn't help me. The first scenario you wrote was "designed" to sound scary :) didn't work as a comparison. The second "scenario" doesn't sound hard at all to maintain within a single logical map (still disregarding any architecture questions).
The interesting thing here is that what we normally do in OO is to build ourselves a good model of the domain and thus through mechanisms like object identity etc we get a normalized non redundant model.
To me it is obvious that there should just be a SINGLE SMPackage called SharedStreams - because there *is* only one such logical package. Then it has multiple releases - and those have attributes. Simple, easy.
I can mark one release as beta. I can describe its dependencies. All straight forward.
And the universe you describe above just turns out to be a *view* (=sub selection of the model based on various criteria). So isn't it actually *easier* to maintain one model together (the Squeak community isn't large enough to maintain many IMHO) and then leave it up to each *user* to select his/her "universe"?
It would be simple to filter out all packages marked as beta or better, for my Squeak version with at least one published realeas available etc. Tada.
I ask about this one issue because if you let go of the idea of having absolutely everything in one index, then the rest of the universes approach seems to mesh very nicely with where Squeak Map seems to be going. It becomes straightforward to let maps be merged into larger maps, and to let non-central maps be locally administered with their own accounts and policies. We could then have simple dependencies, stable releases, mixin servers, private servers, and local update policies, all without requiring any further cleverness.
Lex - you make it sound sooo easy! But it isn't. I do want to investigate how my "tree model" perhaps could be turned into a model where maps can be mixed more freely BUT... it comes with lots of effects:
If you look at the current SM model there are various crosslinks in the model, just a few examples:
1. The categories. 2. The co-maintainers of a package. 3. The publisher (which maintainer published the release) of a release.
...and coming:
4. Resources attachable to other SMObjects, especially: 5. Configurations which is a resource owned by another maintainer than the owner of the release it is attached to - and it refers to other releases owned by other maintainers.
...and perhaps a few I have forgotten.
Now... if the above crosslinks weren't there at all - if the SM model was just a bunch of independent object "trees" - then merging them would be trivial, because the information would be "independent".
Now that is not true, and I have said this over and over. Julian told me that, ok, but the "forreign" links - can't they simple be encoded as the UUID so that they can be ripped out, put on another server and then merged back together and then they get resolved? Hehe... sure. And what happens when the thing it refers to isn't there anymore? etc. etc.
In short - I am saying for the 11th time: Splitting an object model over multiple servers and then somehow magically being able to modify them in a distributed fashion and just merging them back together... it is a hard problem.
I am not saying it is impossible - because that would be stretching it. But it takes a lot of code and it will create lots of situations with "dangling pointers", copies out of synch etc. etc. A lot of decisions to be made.
Now, instead of going over and over like a broken record, this is what I plan to do:
0. Work out (and deploy) the dependency model so that the model can "land" properly. It affects all this a LOT.
1. I have an idea on how to be able to distribute the map on multiple servers with an update/commit model on them and make that work with a reasonable effort. It involves giving each SMObject its own transaction counter etc.
2. We could possibly drop the simplification I made with a tree of servers (only one parent) and try to make it work with multiple parents. It will involve lots of resolve/merge logic I guess.
But I am very afraid of the effects when people start setting up their own little maps all over the place with lots of duplication, redundancy, synch problems, servers being down/up, servers simply being unknown etc... well. Obviously this doesn't scare you at all, which I don't understand why. These are the things I can already hear:
"Hey! Where did you find that? Oh, I didn't know about *that* server.... Hmmm, it isn't up now, do you have a copy you can email me?"
"Hi! I just posted my little Application on my own map *here*. Bye!"
"Hmmm, does anyone have a list of all known maps at this point in time? Sure, here is my list, but I heard that Ned has a bigger list with more stuff on it."
"How many packages are there for Squeak? Well, we don't really know. Ok, but can someone tell me where to look to find ZZZ? Sure, you can look here, and here, and here, and perhaps over there..."
I mean - come on! Am I the ONLY ONE afraid of these scenarios? Am I the ONLY one who remembers how it was before SM? Am I the only Debian user who has been hunting for .deb packages on web sites, looking around for entries to put in my sources.list, wondering where to find package XXX that actually works on Debian?
All in all it is very, very simply. We need ONE model. We need ONE map. No, it doesn't have to be centralized, it can work just like DNS or whatever. Yes, you can have your own little server, just like with DNS.
Yes, it is a pain to write the code so that SM gives us all this.
Finally, let me be very clear on one other thing: all of the efforts on things like dependencies, interfaces, compatible updates, branching versions, and simultaneous loading of multiple versions, are still very important no matter what we do in the short term regarding package catalogs. The present discussion is just about what the next step or two should be.
In summary, I would be happy to have one tool or toolset we all agree on using. But while the really great Grand Unified Tools are still being developed, shouldn't we use simple existing solutions and make do as well as possible?
Of course we should. I just don't see why you portrait yourself as the proponent for something "simple" and "existing" when in fact SM is the thing that exists and is simple. :) Really.
Regards,
Lex Spoon
regards, Göran
PS. Obviously we didn't get much closer to each other in these last postings. I am aware of the fact that there are needs to be met, like private maps and a working dependency model etc. I am focusing on dependencies first - because that is what adds most to the *model*.
Hi Göran--
I think a better analogy is with IRC, not DNS. We could set up a relay network of servers that share information about where to find things. In particular:
"Hey! Where did you find that? Oh, I didn't know about *that* server.... Hmmm, it isn't up now, do you have a copy you can email me?"
We could avoid that: having found any server in the network, a user would find out which other servers provide access to the desired artifact. Artifacts can be mirrored on multiple servers, so one server being down doesn't necessarily pose a problem. The analogous IRC situation is looking for another user; each server can provide information about how any user in the network is connected.
"Hi! I just posted my little Application on my own map *here*. Bye!"
The server that person used could join the server network, providing visibility as described above. If the community cares about protecting against servers going down, then someone will mirror the application on one or more other servers.
"Hmmm, does anyone have a list of all known maps at this point in time? Sure, here is my list, but I heard that Ned has a bigger list with more stuff on it."
That would be a manual recapitulation of what the server network already did automatically.
"How many packages are there for Squeak? Well, we don't really know. Ok, but can someone tell me where to look to find ZZZ? Sure, you can look here, and here, and here, and perhaps over there..."
Again, the server network can tell you where things are.
I mean - come on! Am I the ONLY ONE afraid of these scenarios?
I'm not afraid of them because they seem quite avoidable.
Am I the ONLY one who remembers how it was before SM?
Now you're setting up a false set of choices. :) There are more of them than just "before SM" (and other things) and SM.
have a good trip!
-C
-- Craig Latta improvisational musical informaticist craig@netjam.org www.netjam.org [|] Proceed for Truth!
Hello Craig,
I think, after your proposal you have *one* *logical* map, *physically* distributed over multiple submaps connected via some network, building a graph of nodes being all at one level (no hierarchy).
This sounds more complicated as using a hierarchical graph like - one logical map at one server, as current; - one logical map distributed over a tree structure of servers (with childs all sharing the same parent content), as planned; - one logical map distributed over a tree structure of servers (like the former), with *mirrors* for some important nodes (e.g. the top node) for improving robustness, as making sense IMO.
In such a hierarchy it is always clear, where to look for *all* the public packages (and their dependencies), seen from the local node; namely at the immediate parent (or its mirror).
Multiple parents with *different* content (in opposite to mere mirrors) seems to be more difficult to realize to me than the former variants, and the complexity of your proposal comes thereafter.
The advantage - to be fair - of such a highly decentralized solution would be, that it could be very robust against technical or social failures in large parts (the internet is an example).
The disadvantage would be, that installing some dependency mechanisms appears very difficult to me in such a scenario, too. For dependencies there has to be some place (the resolver), where all needed ones are available at a whole! And not all of them are to be coupled to a certain package: e.g. conflicts of some configurations may be detected later and be stored aside the packages. In addition it probably makes sense to have some control about who is allowed to change the dependencies, which also seems to be more difficult to reach in a fully decentralized model than in a hierarchical one.
BTW: The security aspects could/should become more important in the future for every chosen solution.
Greetings Stephan
Craig Latta wrote:
Hi Göran--
I think a better analogy is with IRC, not DNS. We could set up a relay network of servers that share information about where to find things. In particular:
"Hey! Where did you find that? Oh, I didn't know about *that* server.... Hmmm, it isn't up now, do you have a copy you can email me?"
We could avoid that: having found any server in the network, a user would find out which other servers provide access to the desired artifact. Artifacts can be mirrored on multiple servers, so one server being down doesn't necessarily pose a problem. The analogous IRC situation is looking for another user; each server can provide information about how any user in the network is connected.
"Hi! I just posted my little Application on my own map *here*. Bye!"
The server that person used could join the server network, providing visibility as described above. If the community cares about protecting against servers going down, then someone will mirror the application on one or more other servers.
"Hmmm, does anyone have a list of all known maps at this point in time? Sure, here is my list, but I heard that Ned has a bigger list with more stuff on it."
That would be a manual recapitulation of what the server network already did automatically.
"How many packages are there for Squeak? Well, we don't really know. Ok, but can someone tell me where to look to find ZZZ? Sure, you can look here, and here, and here, and perhaps over there..."
Again, the server network can tell you where things are.
I mean - come on! Am I the ONLY ONE afraid of these scenarios?
I'm not afraid of them because they seem quite avoidable.
Am I the ONLY one who remembers how it was before SM?
Now you're setting up a false set of choices. :) There are more of them than just "before SM" (and other things) and SM.
have a good trip!
-C
-- Craig Latta improvisational musical informaticist craig@netjam.org www.netjam.org [|] Proceed for Truth!
Hi all!
(back from China and will have limited time the next weeks)
Stephan Rudlof sr@evolgo.de wrote:
Hello Craig,
I think, after your proposal you have *one* *logical* map, *physically* distributed over multiple submaps connected via some network, building a graph of nodes being all at one level (no hierarchy).
Yes, this is my perception too. The discussion is going around in "circles" here.
I said "let us stick to ONE logical map, although with an architecture of *connected* multiple servers offering public/private additions etc".
Lex says "let's have multiple *disconnected* servers with MULTIPLE maps that the clients can merge together like they please". My response is that such a model is bad because suddenly we have lost the ability to enumerate the full information space because just like in Debian there is no "map of the repositories". And I have also expressed multiple other worries about the complexity of it all, yadda yadda.
Then Craig says "but we can connect all those servers in a network" - aha! But that is what *I* have been saying all along! :) Although I am trying to make it vastly simpler with a tree instead of a graph.
This sounds more complicated as using a hierarchical graph like
- one logical map at one server, as current;
- one logical map distributed over a tree structure of servers (with
childs all sharing the same parent content), as planned;
- one logical map distributed over a tree structure of servers (like the
former), with *mirrors* for some important nodes (e.g. the top node) for improving robustness, as making sense IMO.
In such a hierarchy it is always clear, where to look for *all* the public packages (and their dependencies), seen from the local node; namely at the immediate parent (or its mirror).
Exactly.
Multiple parents with *different* content (in opposite to mere mirrors) seems to be more difficult to realize to me than the former variants, and the complexity of your proposal comes thereafter.
The advantage - to be fair - of such a highly decentralized solution would be, that it could be very robust against technical or social failures in large parts (the internet is an example).
The disadvantage would be, that installing some dependency mechanisms appears very difficult to me in such a scenario, too. For dependencies there has to be some place (the resolver), where all needed ones are available at a whole! And not all of them are to be coupled to a certain package: e.g. conflicts of some configurations may be detected later and be stored aside the packages. In addition it probably makes sense to have some control about who is allowed to change the dependencies, which also seems to be more difficult to reach in a fully decentralized model than in a hierarchical one.
Again, exactly.
Now... I am dead tired of arguing all this. :) I need to code and make things happen.
I am aware of a number of "use cases" that Lex wants to enable and that are harder in my model. I intend to really try to enable those - the most important one being IMHO to be able to have non-public SMObjects on non-public servers and perhaps even be able to somehow combine multiple such non-public servers.
But I still don't buy the "universe" idea. There are various reasons for this. One major reason is that I don't think we - the community - have the energy nor the actual need to maintain multiple universes that Lex is describing.
The lack of energy is pretty obvious - we even have a hard time maintaining the Full assembly. :)
And what about the needs? We have been comparing with Linux distros a lot but I don't think the comparison should be taken too far. Because I think that we are making a mistake if we think that Squeak has the *same* issues.
For example - most of us I think use Squeak to build applications. We don't use Squeak as an OS. We frequently construct images targeted at certain tasks. I have myself multiple images for different purposes - and often I run 4-5 of them concurrently. One for developing SM. Another for using BFAV. One "throw away" for playing with packages from SM. Yet another for developing a web app, and so on. Each image I taylor for the task. I install a selected few pieces, a few tools I like, a few "libraries" etc trying to keep each image pretty clean.
Now - does this sound like how I use Linux? Nope. Sure, I build targeted servers with Debian too - but that is not what I do "daily". :)
My view is that we are *primarily* a community of package *developers*. We write code and then we publish that code in releases. We have a pretty good idea about the dependencies of the code we have written and we can take the time needed to describe it - if there is a mechanism for doing that.
We are in general not a community of people that like maintaining distros - again, just look at how few of us that actually do the work for maintaining Full etc.
My conclusion from this is that we should strive to describe our packages and their releases as good as possible (including dependencies etc) *individually* and then let the user "pick and choose" among them in a semi-automated controlled manner based on those descriptions.
Or putting it another way - it seems to me our tools and our model should be focused on helping the user to compose an image from packages instead of fooling ourselves to believe that we will have the energy or interest to compose "distros" for others to use.
Well, I may be right or wrong. :)
regards, Göran
goran.krampe@bluefish.se wrote:
But I still don't buy the "universe" idea. There are various reasons for this. One major reason is that I don't think we - the community - have the energy nor the actual need to maintain multiple universes that Lex is describing.
I never suggested maintaining multiple universes as a community, unless you mean the multiple stable releases. While the conversation never got far enough, my working model has always been that there would be a main development universe, plus frozen stable universes that get put out ever year or so. This organization would parallel the status quo, where SqueakMap plus updates are used for development, and gamma images are used for releases.
-Lex
Hello Göran and all,
goran.krampe@bluefish.se wrote:
...
Also - I don't agree fully with Stephan on his model, but I still don't want him to go off and roll his own.
I do *not* want to roll my model on my own. This is *not* possible! A package dependency resolver has to be accepted by the community to be able to work. And both its functionality and its front end have to be understood (not necessarily all technical details).
But of course I try my best to make arguments for my model and to convince others that it is worth trying it. If I cannot show, that it is worth to go with compatibility codes with subset property ordering as explained in http://minnow.cc.gatech.edu/squeak/3792#Subset%20property%20ordering , my model has no chance (then just some backend parts could be of interest for other models, but I don't know if I would work further at the backend in this case). Note: the exact number of compatibility levels may change, but the mentioned subset property ordering is essential.
Parts of the DEPS *backend* (e.g. Transformers) 'just' have to work, and *only* in this area I'm on my own currently (staying in the conceptual phase, improving and detailing concepts (I've underestimated the complexity for a flexible solution there somewhat (there are different kinds of requirements!))). Later - after an agreement regarding the compatibility measurement has been reached! - there will be interface issues with SM, but I don't fear not to find a solution with Göran.
It is better to be able to complement each other and work together.
Fully agreed!
And to enable SM to be a base for it.
Göran, thanks for your work in this direction and your collaborative spirit!
Greetings Stephan
...
squeak-dev@lists.squeakfoundation.org
-
Craig Latta -
goran.krampe＠bluefish.se -
Lex Spoon -
lex＠cc.gatech.edu
-
Stephan Rudlof