Hi Goran and Lex, Hi other Squeakers,
there are following some comments about how my DEPS model could work into the light of some of the thoughts from the BIG posting.
Note: in the following DEPS means the corresponding paper contained in the mail subjected [DEPS][PAPER] Dependencies for Squeak sent to the list recently.
goran.krampe@bluefish.se wrote:
Hi Lex! ...
lex@cc.gatech.edu wrote:
Hey Goran, ...
...
Btw, you did stumble upon a slightly interesting fact though - you said "how do you fit post-release bug reports into the world view?". Given that in the planned SM the dependency information is not embedded in the actual release (as in Debian) this means that the dependency information can be revised post-release. And it should, especially by adding more working know configurations. If the maintainer or anyone else discovers that, darn - there was actually a serious bug in Z1.2 and Z1.3 is really needed - then the configuration can and SHOULD be changed.
This could be modeled as updated Transformations in DEPS.
And if someone discovers that, hey "I used Z2.0 and it seems to work fine with that too!" then that should/could be added too. That is one of the key points of NOT embedding dependency information inside the releases.
This could also be modeled as changed Transformations: see section "Technical terms -> Caps -> Boolean logic".
Aside from the issues with the "guarantees" idea, it has not been established that the jigsaw puzzle of version-specific dependencies is going to be practical for tools to work with. This is a significant technical problem that versioned dependencies add, and everyone seem to be waving it off as something to handle in the future. I don't know that this is solvable, guys, and no one has made a convincing argument otherwise.
I see it more as a *social* problem, to have responsible maintainers trying to do their best. But if a maintainer wants his/her package to be in widespread use - which would give some happyness, at least for me -, then I think he/she is doing his/her best - to make it compatible with other packages other people are using, - to be careful with a potential 'stable' categorization, - to define its dependencies carefully.
And afterwards it is no problem to add conflicts...
Another idea: there also could be some rating system for packages, marking white and black sheeps!
...
dependency jigsaw cannot be adequately solved. Let me run you through a small example. Packages A and B depend on Collections:
A 1.0 needs Collections 1.0 B 1.0 needs Collections 1.0
Fine so far, I install all three packages. Now the collections library gets updated to 1.1. I cannot install the upgrade without breaking the dependencies of A and B!! So, I wait before installing it, even though
I have treaten this example (further parts are snipped here) in DEPS.
Not entirely true. First of all - the engine should be parameterizable (cool word) by you.
So the idea is that if you - at this point - select Collections 1.1 and press "install" it could say something like "The installed package releases A1.0 and B1.0 are only known to work with Collections1.0, but since Collections1.1 is marked by the maintainer as being 'Code changes, but only big fixes' they could still likely work, do you still want to proceed and install it?"
The proposed versioning number system in DEPS is just a model to express such marks like 'Code changes, but only big fixes' technically. Though I more have had *small* fixes in mind as an example for changing the 4. version number...
And if you set a parameter, let me fantasize here - "Allowed compatibility threshold" or something - then perhaps it wouldn't even ask if the new release was below a certain compatibility level.
Again I think it is important to mention the social aspects here: as better the package maintainers categorize their packages, as better such a technical solution could work, if there is only this thing. But there may be more to help the technical solution computing good results, e.g. prefiltering of packages after some criteria: - stable/unstable, - rating by users, - rating by automatically testing, - trust measure of package maintianer (somewhat delicate), - etc..
Now, it doesn't end there. Not only can you install Collections1.1 (and as I described in a much more informed way than if the recorded dependencies were just "A and B needs Collections") but if someone ELSE has already tried this, and discovered A annd B works just FINE (who knows, perhaps there is even a test suite to run!) then he/she can have beaten the maintainers of A and B respectively and attached new configurations to A and B that says that, according to him - A1.0 works just fine with Collections1.1 (and the same for B).
In DEPS to be modeled by Boolean logic in requires.
So if you then trust this person, you can go ahead with even more information to guide you - he said it worked, and since it was Dan Ingalls - you decide to trust him.
This is an interesting point: rating 'configurations' after the people making them. Note to 'configurations': in DEPS these would be expressed as Transformations expressing the dependencies, possibly enriched by a pointer to the respected person together with its *installed* packages to get the whole thing. In other words: who has used which Transformations to get its fancy system configuration.
And my third final point - if there was no Ingalls-config available, and you did have to "take a chance" - then YOU can be the good citizen paving the way for others to follow by verifying it works and attach a new configuration to A and B.
But its expressiveness is limited, if the knowledge about *all* the installed packages hasn't been recorded, too. YOU can have a configuration with very few packages totally incompatible with many others in widespread use, for example.
Now - IMHO this is a GREAT model :). I am not saying we will not discover details to tweak with it - for example, I can surely see it extended with information about "intended dependencies" - meaning that the maintainer might want to tell people what packages he wants to depend on, this may not always be the same thing.
In DEPS the "intended dependencies" would partly modeled by the compatibility level expressed by the version number given with the package. The other part would be Transformations for installing it, first given by the maintainer in addition to the package (there may be some process, too...). Later it should be possible to change the Transformations: normally they would be enriched with conflicts. I currently don't think it makes much sense to change a given version number without changing the package, too, but there may be exceptions (I think this will rarely be used in Debian for overriding the normal dependency mechanism).
I might be the main developer of package C, which also uses Collections. Okay, so eventually tthe author of A, being a great citizen, upgrades their Collection package and reruns their tests -- shock, nothing brock. While they are at it, they add a few class comments, and then post A 1.1 which depends on Collections 1.1. Drats, I still cannot update my Collections library, because that will make me uninstall B.
No big difference, at this point - select Collections 1.1 and press "install" it could say something (quite similar) like "The installed package release B1.0 is only known to work with Collections1.0. A1.0 on the other hand can be upgraded to A1.1 which is known to work with Collections.1.1. Since Collections1.1 is marked by the maintainer as being 'Code changes, but only bug fixes' B1.0 could still likely work, do you still want to proceed and install Collections1.1 and also A1.1?"
Of course, exactly how the engine will talk to the user is a UI issue. :) And upgrading A1.1 would of course be optional - the engine should only be HELPFUL - not forceful. You should always be TOTALLY FREE to install what the heck you like.
This is an interesting point: how to *override* the automatically chosen solution? In DEPS a conflict is a conflict: it makes no sense to override a known conflict. But what about a weak conflict? E.g. Foo_3.2.1.1.1 *may* break Bar under some not very likely circumstances.
Straightforward idea (after some moments of thinking): classifications of conflicts and using them together with the user install policy as a filter together with hints to the user.
More simple solution: not stating the conflict in the DEPS-Transformation, but writing a warning (possibly standardized, so that GUIs could issue it automatically) into the description of the package. Then the user has the choice...
...
As I have clearly shown this is not the case. I have NEVER argued for the dependency engine to be some kind of POLICE. You can still install *whatever you like*. The whole point is that you can do this and be AWARE of what it means and what the risks are.
Additional idea: policy ignoring conflicts in DEPS-Transformations as option to force an install.
And I have also, I hope, clearly shown that since we can all attach tested working configurations you will not be stuck at the whim of the maintainers.
And I also hope i have shown that the configurations can be added AFTER the release,
I think of introducing some whatever dependency system slowly: at first only a few packages get their technically usable version numbers. Then the first package is published with separately delivered DEPS-Transformations describing its dependencies to these few packages, so they can be automatically installed with it. This is so comfortable that more and more package maintainers jump onto the same train...
and even MODIFIED after the release.
This is a crucial point: a package should contain its expected compatibility level (via its version number), but not its dependencies.
This is a GOOD THING - because if they are wrong they should simply be fixed.
Agreed.
...
Of course, it all gets "worse" in some kind of way with lots of intertwined packages - but I don't think the model will suffer. There are some really nice things here helping it:
- Users can help keep the dependencies tracked and verified, not only
maintainers. This is cool.
Here again I'm seeing some social questions: who is responsible to change the 'official' dependencies which the non expert (or an expert not knowing the 1000th package) user of Squeak *normally* uses? Since he/she wants to have the one-click install...
- Pressure can build on key packages that really needs to be upgraded
in order to work properly.
This leads me to something I haven't talked about before - it should be possible to record an "anti configuration" - meaning that "No, sorry, I tested A1.1 with Collections1.2 and nope, didn't work". This would be invaluable to have. Again, we record information - that is all. Then how the information is used is purely up to you. I can even imagine different engines or at least clearly different pluggable strategies. :)
Recording the "anti configuration" makes sense.
But I think while writing DEPS I've had a more rigid policy in mind. Assumed the observation above is true: then it is a *conflict*, isn't it? Which *normal* user not trying to fix the bugs (say > 99% of all users) would ever like to install an "anti configuration"?
So this "anti configuration" should be expressed as a conflict to protect the normal user. For the few experts, which are not only experts, but also interested in this special broken "anti configuration" the override buttons should be there (but switched off in the official image).
acceptible that every package needs to be tested with each incremental version of every other package, and thus achieve the lockstep progression? That seems very inefficient if most package updates are compatibility-preserving bug fixes.
I have described that the compatibility level categorization of releases will make this information explicit.
If you don't want to test your releases at all - then fine, others will do it for you. :)
The maintainer has made at least a minimal test, if there has been just one run of the package...
To an automatically dependeny resolving system: I think such a bleeding edge package should just been outside automatically installation, since it would make more problems than it helps. A few of such packages and you can start with a fresh image... The good thing: no responsible maintainer of another package would put such a bleeding edge package in the requires of his/her package! So possibly this is already the automatical solution of this policy problem given without any costs...
And if you do test them, then why not record which releases you tested it against?
This certainly makes sense.
...
And it will also be very suitable for test driven development. Did you hear that Stephane? :) Because if we can attach or in some other way associate how to run the tests with the packages (included or not) then they will form a formal basis of testing out new configurations of releases.
Hell, it could even be automated in theory - we could have a robot server sitting trying to test new combinations of package releases and attaching those autotested configurations automatically.
This would be very nice!
To the questions of single or multiple tree or not tree'd maps -------------------------------------------------------------- A dependency system should work with distributed maps as well. There should be distributed dependencies then, too; in the tree case: a local server surely has a map containing some local packages (why to have it otherwise?), and therefrom it has at least some dependencies between packages to be expressed, which its master doesn't know, if a dependency system should be used for these local packages.
... <snipped very much here>
Greetings Stephan