Dear Squeakers, Dear Goran,
at first a serious warning: the draft for this mail has undergone some iterations and finally it has become a
*** MONSTER mail! ***
So please snip away the not affected parts, if you reply.
In spite of being a MONSTER mail, I hope it is a start for fruitful discussions.
Greetings, Stephan
####################### Dependencies for Squeak #######################
Contents ########
- Contents - Foreword - To the reader - Overview - Technical Terms - Caps - Transformations - To Blocks as input - Boolean logic - Outlook to packages - How to model dynamic provides? - Note - Actions - Algorithms - Application to packages - Packages - Versions - Classification issues - Note to fixes - Automatically upgrading - Why major version number 1. at all? - Consequences - Installation of packages - Different kind of installs - Examples (for looking, if the concepts are working) - Package WebBrowserStandard - Upgrades - Upgrade by deletion/install - Incremental upgrade - Make tool - To Lex's example - Relationships with existing tools and concepts - SM - Monticello - Class extensions - Missing Stuff - Roadmap (loosely) - Personal remarks
Foreword ########
Triggered by a recent help request regarding dependencies from Goran, I've started thinking about the issues around and this is the result...
Some time ago I've made some stuff in direction of a Meak (make for Squeak), its importance has sunken after the release of VMMaker.
Now my idea is to make a very general dependency mechanism, which should be usable to build both a Debian apt-like package system, and a make tool as well. Some of the logic is similar.
This paper - I think it's OK to call it so - should not be seen as a rock solid concept paper: it has undergone some iterations so far, but I have consciously stopped thinking further at this point to look for reactions!
To the reader =============
I know, it takes some time to read through this paper (it also has taken some time to write it ;-) ), but I hope you will find some interesting thoughts in it.
Overview ########
If you are not interested in the technical, but in the social issues, you nevertheless should read section 'Application to packages' and especially its 'Application to packages'->'Versions' subsection!
Otherwise please take a look into 'Contents' or just continue reading from here...
Technical Terms ###############
Caps ==== There are Capabilities (Caps) each describing the capability to do something. Caps mostly depend on other Caps, which are prerequisites of them. In the following Caps means Capabilities or a union of them.
Transformations ===============
Transformations describe how it is possible to change a set of Caps: they may - virtually at first - add or remove Caps or both. They transform one set of Caps into another one. To do their work they perform Actions.
Inputs are I.1. requires: required Caps, and I.2. conflicts: conflicting Caps, outputs are O.1. provides: newly generated Caps, and O.2. deletes: deleted Caps. Properties: - provides are const sets, but - deletes are configuration dependent.
Outputs are expressed as - set of (named) Caps.
Inputs are expressed as 1. set of (named) Caps (as for outputs), or 2. as a Block - taking a Caps set as an argument, returning - true, if the requirement/conflict has been fullfilled, - false, otherwise.
To Blocks as input ------------------ It is computational expensive to use Blocks as input, since in principle all existing Caps and those somehow generatable by whatever Transformation combination (!) have to be checked.
An - important, I think - improvement for the case of expressing conflicts may be to divide the Blocks into two classes (later there may be more): 1. Blocks just checking conflicts with Caps generated additionally by the packages generating the requirement Caps, under the precondition, that the requirement Caps are given as set. 2. Other Blocks.
See section "To Lex's example" below for a motivation of Blocks as arguments here and the suggested improvement.
Boolean logic ------------- Require sets model a logical AND of requirements. Logical ORs - and therefrom Boolean logic - can be modeled via additional Transformations with newly introduced Caps names. This is far more efficient as trying to do this in one Transformation using the Blocks mechanism!
Outlook to packages ------------------- A package removal can be modeled by transforming the all over package Caps as requires into deletes of this and all other package related Caps.
How to model dynamic provides? ------------------------------ The restriction 'provides are const sets' eases some things, but it is not a limitation AFAICS.
Example: Installing a package WebBrowser may result into more Caps if some other package SuperFonts is there. This can easily be modeled by a *logical* (no Action) package Transformation describing the all over functionality of the packages combo, with - requires: Caps ('WebBrowser', 'SuperFonts'), and - provides: something like ('WebBrowserWithSuperFonts').
Note ---- It's questionable, if naming both set and Block inputs the same is a good idea, but I want to stop for now...
Actions ======= Actions are commands leading to a change of the (set of) Caps in the system. If triggered, they add and/or remove one or more Caps in reality.
Algorithms ############
Before there is no good model of the domain, it is not urgent to think about algorithms. I don't fear not to find some...
A few things I'm currently expecting to be faced with: Easy: - depth first traversing of dependency graphs, - back tracking with detection of cycles. Not so easy: - combinatorial explosion, if Blocks as inputs for Transformation requires and conflicts come into play (but section 'Transformations->To Blocks as input' contains a first idea for embanking it).
Application to packages #######################
Packages ========
A Package provides Caps: these are representing some things to be made with or by the Package (interfaces, commands). There may be more than one Caps per Package!
A Package needs Caps provided from other packages (requires).
A Package providing some Caps may conflict with another Package (providing other and/or the same Caps). To be able to express these conflicts at Caps level it is necessary that all packages are mutually different in at least one Caps! But this is easy to accomplish, if there is just one Caps named so, that it contains the - unique - name of them.
A conflicting Package has to be removed (this can be difficult) before installing a new one, leading into a reduction of its provided Caps (there may be logical ones left, which are also provided by another package). There may also be Packages, which only remove Caps without adding new ones (system shrink)!
Versions ========
This section has become very long and is very crucial regarding its social aspects.
There *have* to be policies to be followed by the package maintainers. I'm currently thinking of version numbers like 1.1.3.4.2 #(1 1 3 4 2) 2.7.3.1.1 #(2 7 3 1 1) , where a change in one of the numbers means:
1. incompatible, changed interface semantics, called major package number: the package is expected to be incompatible with older or newer ones having another number here (at this position);
2. downwards compatible, called minor package number: the package has changed, and their may be new features, but it is expected to work with software written for lower numbers here (but with same 1. number, of course);
3. downwards *and* upwards compatible: the package has changed, but there are *no* new features here (but it may be faster for example), so it is expected to work with software written for higher numbers here (but with same 1., 2. numbers, of course);
4. compatible: 'small' fixes without changed functionality in the sense above (also see 'Classification issues' below);
5. no code changes at all: no changed semantics, but e.g. improved class comments.
After this scheme, a version number a.b.c.d. for a package P corresponds to the Caps 'P', 'P_a', 'P_a.b', 'P_a.b.c', 'P_a.b.c.d', 'P_a.b.c.d.e'. Caps 'P_a.b' - normally - includes *every* Caps 'P_a.x', x < b, since the functionality/interface has just been smaller then. The longer numbers are for expressing conflicts and requirements in a world not being perfect. This means, we finally have defined a relationship between Caps and version numbers! And it should be clear now, why there is this funny plural with the 'Caps' term: it is a normal case for a package-version combo, that it provides a *set* of Capabilities!
Classification issues --------------------- It seems to be difficult to classify fixes (4.) correctly. Fixes definitely change the formal semantics, so one has a good reason to rank them higher and exchange them with 3. for example. A version number change in 3. normally doesn't change the formal semantics (but may introduce new bugs as a change in 4. as well, changing the formal semantics then).
But if a 'small' fix just makes the package working with some seldom used other package being the only one running into the bug, then the classification as 4. makes sense (3. may introduce new bugs affecting many packages). But if the fix removes some serious problems affecting many other packages, it may as well classified as 2., since there is good reason to view this as an important upgrade.
A similar argumentation could be made for changes in the functionality classified as 3.: a piece of code just running much faster without changing the formal semantics, could open up new use cases. This may as well earn a classification in 2..
Currently I think the intelligence of the package maintainers is questioned here, since 'it depends'...
Note to fixes ------------- An important fix (changed version number 4.) can make the package work the first time with another one. It may make sense - especially if the major number recently has changed - to apply such a fix to an older version, too! This would lead to a new older package with a new number, of course.
Automatically upgrading ----------------------- In general automatically upgrading should - apply to all packages increasing their minor or lower ranked version number, - *not* apply to packages increasing their major version number.
A harvesting process could be introduced using this technical solution as follows: addditional classifications of packages as stable/unstable/etc. could prefilter the number of packages to be searched for fulfilling dependencies (this classification could also be realised by just putting them into classified - different - package directories).
Why major version number 1. at all? ----------------------------------- After this versioning policy automatically upgrading would (mostly) work from version to version with a *fixed* first version number. If the major number changes, there are incompatible versions (since we have defined it so!). This could lead to the question, why not to ommit this major number and to force the package maintainer to publish a new package with a different name instead.
Arguments for a major version number increase instead of a name change:
- There are cases for expressing the existence of some functionality at a very high level independent from the major number: e.g. some package wants to have a package WebBrowserStandard for enforcing to give the user a chance of viewing its generated HTML files (without assuming any API for calling it itself): a newly created name (how to guess it for other package maintainers?) instead of a major number increase would make expressing this dependency impossible.
- Though an interface has changed, the functionality may be very similar, and the package name should express functionality.
- The package name combined with the major number could be seen as the - technically and socially - needed new name.
Consequences ------------
This is a client POV versioning policy. There may be others, of course: IMHO it's not so important which policy to have, but
*** It's very important to have a policy at all! ***
Otherwise a technically working dependency mechanism doesn't work for packages because of social reasons...
Technically this means to define some Caps <-> version number relation with an order of Caps corresponding to an order of version numbers like described above (or to use a similar mechanism).
*** Otherwise the technical solution *cannot* work! ***
Installation of packages ======================== In general there are - pre-install, - do-install, - post-install, - pre-remove, - do-remove, - post-remove, Scripts (they may be empty). They are triggered by performing an (package install/removal) Action.
Normally the install phase results into adding all provided Caps and the remove phase into the removal of them. Potential conflicts have to be solved *before* installation!
Different kind of installs -------------------------- There are different kind of installs: - upgrading, - full installation with a previous removal of an older version (if it exists).
Examples (for looking, if the concepts are working) ##################################################
Package WebBrowserStandard ==========================
- Provides Caps 'WebBrowser' and more; - needs Caps 'SocketStuff' and 'RenderingMachine'; - exists in versions 1.1.3.1.7, 2.1.2.3.1, 2.2.1.1.1; - conflicts with package 'WebBrowserNicerButSlow' and some special SocketStuff versions.
Each package version combo WebBrowserStandard_1.1.3.1.7, WebBrowserStandard_2.1.2.3.1, and WebBrowserStandard_2.2.1.1.1 needs a Transformation describing pre- and post Caps sets.
E.g. for WebBrowser_2.2.1.1.1 there is the following:
a) It provides Caps 'WebBrowserStandard_2.2' being a super set including all Caps provided by WebBrowserStandard_2.1.2.3.1, but *not* all Caps provided by WebBrowser_1.1.3.1.7 (since the major number has changed);
b) It may require Caps ('SocketStuff_3.4', 'RenderingMachine_2'). A package RenderingMachine_2.1 would provide the last requirement.
c) It may conflict - with package WebBrowserNicerButSlow in general (since it uses the same sockets); and - with SocketStuff_3.4.3.7.x (arbitrary x), since the bugfix there had some very special side effect - unfortunately another installed important package PWorkaround needs exactly one of this versions of SocketStuff: it made a workaround not compatible with a newer version of it, where the side effect has been removed... (world is not perfect).
Note: if the user wants to install WebBrowserNicerButSlow or any other package, all packages have to be checked for conflicts.
How to model c)? - The conflict with package WebBrowserNicerButSlow is an all over package conflict and will be modeled as a conflict with Caps ('WebBrowserNicerButSlow'), which includes all Caps ('WebBrowserNicerButSlow_a.b.c.d.e') for arbitrary a, b, c, d, e. - The conflict with SocketStuff in versions 3.4.3.7.x is *not* an all over package conflict, it is a package-version conflict. This will be modeled as a conflict with Caps ('SocketStuff_3.4.3.7').
d) It has a Transformation requires: ('SocketStuff_3.4', 'RenderingMachine_2.1') conflicts: ('SocketStuff_3.4.3.7') | V provides: ('WebBrowserStandard', "all over package Caps" 'WebBrowserStandard_2', "major release" 'WebBrowserStandard_2.2', "minor release" 'WebBrowserStandard_2.2.1', 'WebBrowserStandard_2.2.1.1', 'WebBrowserStandard_2.2.1.1.1', 'WebBrowser' "logical Caps"). The last Caps in the result is a logical Caps provided by both packages WebBrowserStandard and WebBrowserNicerButSlow. Such a beast makes sense for expressing dependencies which shouldn't be bounded to a certain package: in this example there may be a point in the system automatically starting the installed WebBrowser, whatever it may be (WebBrowserStandard or WebBrowserNicerButSlow).
The question in case of holding package PWorkaround is, if there is a version SocketStuff_3.a.b.c.x with a >= 4, b, c, x arbitrary, but not ((a = 4) & (b = 3) & (c = 7)), which also gives the input Caps 'SocketStuff_3.4'! Otherwise it wouldn't be possible to install WebBrowserStandard without removing PWorkaround first. So the package-version conflicts are reducing the number of packages capable of providing the needed input caps.
Upgrades ========
An upgrading of WebBrowserStandard_2.1.2.3.1 to WebBrowserStandard_2.2.1.1.1 may be modeled by - a deletion of the package WebBrowserStandard_2.1.2.3.1, followed by - an installation of WebBrowserStandard_2.2.1.1.1, or by - an incremental upgrade.
Upgrade by deletion/install ---------------------------
requires: ('WebBrowserStandard') | V deletes: ('WebBrowserStandard', "all over package Caps" 'WebBrowserStandard_2', "major release" 'WebBrowserStandard_2.1', "minor release" 'WebBrowserStandard_2.1.2', 'WebBrowserStandard_2.1.2.3', 'WebBrowserStandard_2.1.2.3.1', 'WebBrowser' "only deleted, if there is no other package providing it").
Note: deletes are depending from the installed version and computed dynamically.
This is followed by a Transformation installing the new version WebBrowserStandard_2.2:
requires: ('SocketStuff_3.4', 'RenderingMachine_2.1') conflicts: ('SocketStuff_3.4.3.7' | 'WebBrowserStandard' "this is the point here!") V provides: ('WebBrowserStandard', "all over package Caps" 'WebBrowserStandard_2', "major release" 'WebBrowserStandard_2.2', "minor release" 'WebBrowserStandard_2.2.1', 'WebBrowserStandard_2.2.1.1', 'WebBrowserStandard_2.2.1.1.1', 'WebBrowser' "logical Caps").
The conflict of Caps 'WebBrowserStandard_2.2' with Caps 'WebBrowserStandard' leads to a search for the deletion Transformation: after the deletion Transformation the conflict is resolved, and after the installation Transformation Caps 'WebBrowserStandard' is back again. This works independently from the installed older or newer version!
Incremental upgrade -------------------
requires: ('WebBrowserStandard_2.1') | V deletes: ('WebBrowserStandard_2.1', "minor release" 'WebBrowserStandard_2.1.2', 'WebBrowserStandard_2.1.2.3', 'WebBrowserStandard_2.1.2.3.1') provides: ('WebBrowserStandard_2.2', "minor release" 'WebBrowserStandard_2.2.1', 'WebBrowserStandard_2.2.1.1', 'WebBrowserStandard_2.2.1.1.1').
Note: the Caps for all over package 'WebBrowserStandard', the major release 'WebBrowserStandard_2' and the logical 'WebBrowser' haven't been changed here.
Make tool =========
A Caps set needed by an executable corresponds to e.g. a couple of object files created from another couple of source and header files.
The important point here is, that changing a required (e.g. a header) file leads to the need for regenerating all files depending on it: - if all the requirement files are older, all is OK; but - if just one requirement file is newer, the target has to be build again.
Each file corresponds to a provided Caps. The semantics of a such a Caps is, that - the file exists, and - it is newer than its required Caps corresponding to files (there may be more). Note: this is another motivation for the use of plural in Caps...
Each not automatically generated file (e.g. header, source files) fulfills the first condition, but not necessarily the second. Dependencies may be modeled - flat, so that the Transformation for each generated file requires all direct or indirect files in the dependency chain (e.g. both headers, if there is a header including another header); - hierarchically, so that the Transformation for each generated file requires only the direct dependencies.
An example for Transformations: requires: ('funnyLib.so' 'fancyProg.o') | action: linking V provides: ('fancyProg')
requires: ('funnyLib.o' 'someSysLib.so') | action: linking V provides: ('funnyLib.so')
requires: ('funnyLib.c' 'funnyLib.h') | action: compiling V provides: ('funnyLib.o')
requires: ('fancyProg.c' 'funnyLib.h') | action: compiling V provides: ('fancyProg.o')
"header to header dependency" requires: ('someSysLib.h') | action: updating time stamp V provides: ('funnyLib.h')
This is a hierarchical modeling of dependencies. For a flat modeling the previous "header to header dependency"-Transformation could be omitted, but then two others providing ('fancyProg.o') and ('funnyLib.o') had to be changed:
requires: ('fancyProg.c' 'funnyLib.h' 'someSysLib.h') | action: compiling V provides: ('fancyProg.o')
requires: ('funnyLib.c' 'funnyLib.h' 'someSysLib.h') | action: compiling V provides: ('funnyLib.o')
The algorithm would be a depth first traversing of the dependency graph and performing the corresponding Action, if the output Caps conditions (file exists and time stamp is newer than this of its requirements) are not fulfilled. In the case of the header to header dependency above, the Action would just be a 'touch' of the file.
A first step before could be to traverse the dependency graph without performing the Actions, to see, if it is possible to build fancyProg at all. This shows the two aspects of Transformations: - describing the dependencies, and - describing the actions to fullfill them.
Remark to packages: before installing a package the first aspect alone is the interesting one to check for conflicts and missing dependencies.
To Lex's example ================
Obviously Lex has argued with another - not set oriented - dependency mechanism in mind. His points are valid with this background.
So I've just used his example - to show how it would work with the mechanisms described here, and - *not* to argue against him!
lex@cc.gatech.edu wrote:
...
Let me run you through a small example. Packages A and B depend on Collections:
A 1.0 needs Collections 1.0 B 1.0 needs Collections 1.0
Fine so far, I install all three packages. Now the collections library gets updated to 1.1. I cannot install the upgrade without breaking the dependencies of A and B!!
After the proposed versioning policy the upgrade would be possible, since Caps ('Collections_1.1') is a superset of ('Collections_1.0').
But here again: we *need* a versioning policy, which people are following!
So, I wait before installing it, even though I might be the main developer of package C, which also uses Collections. Okay, so eventually tthe author of A, being a great citizen, upgrades their Collection package and reruns their tests -- shock, nothing brock. While they are at it, they add a few class comments, and then post A 1.1 which depends on Collections 1.1.
"Transformation for A 1.1" requires: ('Collections_1.1') | action: installing V provides: ('A_1.1', ...)
Drats, I still cannot update my Collections library, because that will make me uninstall B. Note that I still cannot upgrade my Collections library, unless I uninstall B.
There is no problem with the proposed scheme here.
But lets introduce an odd incompatibility: Transformation for B 1.0 requires: ('Collections_1.0') conflicts: ('Collections_1.1') | action: installing V provides: ('B_1.0', ...) , which is a violation of our policy; but hey, shit happens. Let's further assume that ('Collections_1.x'), x >= 2 would be OK again.
This would trigger a conflict if trying to upgrade to A 1.1, which is the wished thing here, since there is a real problem then. But after Collections 1.2 being out, the A 1.1 upgrade would work, since then dependency resolver would just use the Transformation to Caps ('Collections_1.2') as a superset providing ('Collections_1.1') as well.
But thinking about this sheds light to a related problem. What about B 1.0 being incompatible to *all* versions of Collections above 1.0? At first this would be a serious violation of the versioning policy. Second, this leads to the feature request of being able to express such an incompatibility as argument to 'conflicts:' (this has led to a change in the definition of Transformations above).
Finally, suppose Collections gets updated to 1.2, and finally B gets around to doing an upgrade. B 1.1 depends on Collections 1.2. Now what? I *still* cannot upgade B, because it depends on Collections 1.2, which is inconsistent with all available versions of A. I end up using A 1.0, B 1.0, and Collections 1.0. And I still cannot update the version of Collections that my C package works with, because I can't install the Collections update anyway. If I'm lucky, package A will release a version that works with 1.2 and I can upgrade everything. But notice: int hat case, every single package in the example has had to *synchronize* their releases, in order to reach a state where the jigsaw has a solution. I could as well be unlucky, and A might declare itself compatible with Collections 1.3.....
It appears that all the packages need to be upgraded in lockstep. I can't offer you a proof for the more general case :), but I imagine it will only get *worse* as more packages enter the picture. Is it acceptible that every package needs to be tested with each incremental version of every other package, and thus achieve the lockstep progression? That seems very inefficient if most package updates are compatibility-preserving bug fixes.
Thinking in capability sets and a versioning policy is needed.
But testing is nevertheless important: it would be very good to have many automatically runnable tests to classify package combinations as stable/unstable/bleedingEdge/etc.. Since we are faced with a combinatorial explosion here, hints regarding stable/unstable/bleedingEdge/etc. from the package maintainers would be helpful though...
...
Relationships with existing tools and concepts ##############################################
SM ==
If the proposed version policy would be used for classifying packages at SM, the release flag of packages would be replaced or modelled by a corresponding version number, which is much more fine granular.
An important question: Who is responsible for maintaining logical packages?
An - already known - example: packages WebBrowserStandard and WebBrowserNicerButSlow may both generate Caps 'WebBrowser' by an installation. There also may be a logical package WebBrowser which just depends on one of them. But who makes the choice? This is similar to the question, who decides which stuff is worth enough to come into the official distribution.
It would be interesting to enrich SM with a dependency mechanism as described in this paper...
Monticello ==========
Monticello versions are *not* package versions as described here! A 'released' Monticello version needs to be assigned a package version number with some policy as described above.
Class extensions ================
It would be nice to have a detection of class extension conflicts. The most simple variant (there are more!) would be to detect, if some installation tries to overwrite an existing class extension method with changed *code* (a changed comment doesn't matter) and ask the user how to proceed. Note: I have had the problem of wanting to use a class extension which another package also had made, without having a dependency from the other package (I just used this small class extension and not the other package). I haven't changed the code (and therefore semantics) of this method (so this has not been a problem *here*); but it was just by chance, that I had seen this class extension from the other package at all (since it was installed incidentally). It's easy to introduce a not compatible class extension without knowing...
While thinking about this, I have an idea: what about a repository server registering class extensions? Then tools could check for a potential conflict with whatever package introducing them...
Missing Stuff #############
Naturally many things are missing here, a few of them are - security considerations: - web of trust of package maintainers, - signature of packages, - data encryption while communicating with the package - directory, - repositories; - Debian-like unstable -> testing -> stable process; - single or multiple package directories (SMs).
Roadmap (loosely, may change) #############################
0. Posting this mail and looking for the reactions.
*** If there is enough interest: ***
1. Creating a Swiki page with this stuff as a start.
2. Getting a consensus for a versioning policy by - getting feedback via ML discussions, - posting an improved draft, for some iterations. To come to a consensus at this point is *very* important, *independently* from the chosen technical solution!
3. Going further with discussions about technical ideas: the concepts should be good *before* coding. This doesn't mean to discuss forever and not to start with coding after a while...
Personal remarks ################
Don't try to be too perfect: KISS! I have to admit that KISS seams to be quite complex in this domain...
I'm willingly to invest some more time - e.g. by coding - into Dependencies for Squeak, but there should be a good chance, that the results won't be thrown into the trash can...
Stephan
What advantage do we get from separating 3 (downwards and upwards compatible) from 4 (small fixes)? Under what conditions would one ever want to accept a version that (supposedly) differs only by small fixes, but not accept a version that (supposedly) is downwards and upwards compatible?
What advantage do we get from separating 5 (no code changes) from 3 and 4? In each of these cases, the author is stating that he or she thinks each version will generate the same results. Either we trust the author or we don't.
I guess some users would trust the author unless another user has sent a configuration report to SqueakMap asserting that the author is wrong for their particular configuration; while other users will not trust the author until some other user has sent in a report that exactly this configuration does work. But I guess bold users will trust the author in each of your cases 3, 4, and 5, while cautious users will not trust the author in any of these cases. Do you disagree? Do you think some users will trust the author when he or she says only comments have changed, but not trust the author when he or she says the code is upwards and downwards compatible?
Stan,
thank you for reading the paper and your questions!
stan@stanheckman.com wrote:
What advantage do we get from separating 3 (downwards and upwards compatible) from 4 (small fixes)? Under what conditions would one ever want to accept a version that (supposedly) differs only by small fixes, but not accept a version that (supposedly) is downwards and upwards compatible?
The idea in my scenario has been, that each improvement - performance, storage, whatever - without changing the call semantics contains some risk of introducing *new* bugs. The 'small fixes' should be viewn as explained in the paper in section 'Application to packages->Versions->Classification issues': * But if a 'small' fix just makes the package working with some seldom * used other package being the only one running into the bug, then the * classification as 4. makes sense (3. may introduce new bugs affecting * many packages).
Directly there follows the relativization: * But if the fix removes some serious problems affecting many other * packages, it may as well classified as 2., since there is good reason to * view this as an important upgrade.
What advantage do we get from separating 5 (no code changes) from 3 and 4? In each of these cases, the author is stating that he or she thinks each version will generate the same results. Either we trust the author or we don't.
If the author is earnest, you can be *sure* that nothing breaks, if there hasn't been any code change (ignoring the possibility of changing the code by accident). But similar to the point above: if there is some code change there *always* is a risk of breaking something. I think it is worth to differentiate between these cases.
I guess some users would trust the author unless another user has sent a configuration report to SqueakMap asserting that the author is wrong for their particular configuration; while other users will not trust the author until some other user has sent in a report that exactly this configuration does work. But I guess bold users will trust the author in each of your cases 3, 4, and 5, while cautious users will not trust the author in any of these cases. Do you disagree?
These behaviors are all possible.
Do you think some users will trust the author when he or she says only comments have changed, but not trust the author when he or she says the code is upwards and downwards compatible?
Yes. But not so hard: 'not trust the author' in a sense that he could have made a mistake or that the software runs into not expected behavior (especially if interacting with packages the author is not responsible for!). And then I think it also depends on the author, if I'm willing to trust him/her more or less, and on the experiences I've made with the package in the past...
I wouldn't expect any *new* problems in case 5. (no code changes at all, stated by a respected maintainer), and therefore assuming a detected bug is an old one; but in every other case I would possibly look into the code changes or just try to install the previous version, if something seems to be broken.
Other points:
- The granularity of the proposed scheme is fine, but better too fine as too simple: nobody is forced to publish versions only changing in lower ranked version numbers, but it is possible. And it is also possible to be very fine granular in expressing conflicts with other package versions: from a singular version to larger and larger sets of versions.
- If the technical solution works in praxis, *strongly* depends on the people using it; in this case especially the package maintainers deciding into which category a package change belongs. In addition there could be some process, of course: web of trust between maintainers, classification in stable/unstable/etc., automatically testing of standard configurations, etc.. But in the beginning I don't expect and I don't like to have too much process: let's have a start first!
Greetings, Stephan
Hi Stephan and all!
I got a bit tired of writing long postings, but since I definitely don't want to loose the opportunity to cooperate with Stephan here I will take the time to reply to this one know.
Before I start - Stephan - don't be offended if I don't agree with you now. :)
And I will try to snip parts that I have no comment on.
Stephan Rudlof sr@evolgo.de wrote: [SNIP]
Technical Terms ###############
Caps
There are Capabilities (Caps) each describing the capability to do something. Caps mostly depend on other Caps, which are prerequisites of them. In the following Caps means Capabilities or a union of them.
This "Caps" layer is an extra abstraction layer that I am not sure about. I understand the theory behind it - but it feels a bit on the overly complex side, at least if it is the primary mechanism. If it was an additional abstraction that one could select to use or not, then it would be another matter.
But I will read on first... :)
Transformations
Transformations describe how it is possible to change a set of Caps: they may - virtually at first - add or remove Caps or both. They transform one set of Caps into another one. To do their work they perform Actions.
Inputs are I.1. requires: required Caps, and I.2. conflicts: conflicting Caps, outputs are O.1. provides: newly generated Caps, and O.2. deletes: deleted Caps. Properties:
- provides are const sets, but
- deletes are configuration dependent.
Outputs are expressed as
- set of (named) Caps.
Inputs are expressed as
- set of (named) Caps (as for outputs), or
- as a Block
- taking a Caps set as an argument, returning
- true, if the requirement/conflict has been fullfilled,
- false, otherwise.
Hmmm, ok - at first look this seems again pretty... advanced. Or complex.
To Blocks as input
It is computational expensive to use Blocks as input, since in principle all existing Caps and those somehow generatable by whatever Transformation combination (!) have to be checked.
An - important, I think - improvement for the case of expressing conflicts may be to divide the Blocks into two classes (later there may be more):
- Blocks just checking conflicts with Caps generated additionally by
the packages generating the requirement Caps, under the precondition, that the requirement Caps are given as set. 2. Other Blocks.
See section "To Lex's example" below for a motivation of Blocks as arguments here and the suggested improvement.
Ok, this part isn't at all clear to me at this point. But it is only a reflection so...
Boolean logic
Require sets model a logical AND of requirements. Logical ORs - and therefrom Boolean logic - can be modeled via additional Transformations with newly introduced Caps names. This is far more efficient as trying to do this in one Transformation using the Blocks mechanism!
Ouch. Ok, my first reaction is "NO!". :) I know it is tempting to add the capability to use boolean logic to express dependencies/scenarios or whatever - and Debian uses it - but personally I have elected to NOT go there because the complexity seems to me to be unwarranted.
But ok, shutting up and reading on... :)
Outlook to packages
A package removal can be modeled by transforming the all over package Caps as requires into deletes of this and all other package related Caps.
Don't understand, but not important at this point I guess.
How to model dynamic provides?
The restriction 'provides are const sets' eases some things, but it is not a limitation AFAICS.
Example: Installing a package WebBrowser may result into more Caps if some other package SuperFonts is there. This can easily be modeled by a *logical* (no Action) package Transformation describing the all over functionality of the packages combo, with
- requires: Caps ('WebBrowser', 'SuperFonts'), and
- provides: something like ('WebBrowserWithSuperFonts').
Ok, at this point I am thinking - perhaps this Caps layer should be something on top of SM? This may mean that your model could be a layer on top of mine. Just a feeling at this point.
Note
It's questionable, if naming both set and Block inputs the same is a good idea, but I want to stop for now...
Actions
Actions are commands leading to a change of the (set of) Caps in the system. If triggered, they add and/or remove one or more Caps in reality.
Algorithms ############
Before there is no good model of the domain, it is not urgent to think about algorithms. I don't fear not to find some...
A few things I'm currently expecting to be faced with: Easy:
- depth first traversing of dependency graphs,
- back tracking with detection of cycles.
Not so easy:
- combinatorial explosion, if Blocks as inputs for Transformation
requires and conflicts come into play (but section 'Transformations->To Blocks as input' contains a first idea for embanking it).
Yes, the reason for me starting to write code was that I wanted to explore at least some of the algorithms involved.
Application to packages #######################
Packages
A Package provides Caps: these are representing some things to be made with or by the Package (interfaces, commands). There may be more than one Caps per Package!
Since this would be a thing changing over time I would presume that actually a package *release* provides Caps?
A Package needs Caps provided from other packages (requires).
Same thing here. In my model this extra "redirection layer" is not present - there a release has tested configurations and each configuration more or less lists the other package *releases* that are needed for the release to work.
Which made me think perhaps this capability layer can be "on top" of my model.
A Package providing some Caps may conflict with another Package (providing other and/or the same Caps).
To be able to express these conflicts at Caps level it is necessary that all packages are mutually different in at least one Caps! But this is easy to accomplish, if there is just one Caps named so, that it contains the - unique - name of them.
Note that a Cap could easily be modelled using SMCategories. That is the point with SMCategory - it is a unique canonicalized "tag" that can be attached to SMObjects. I am not sure what you are saying though in the last part. Are you saying that two different packages (releases) can not provide the exact same Caps? Why not?
A conflicting Package has to be removed (this can be difficult) before installing a new one, leading into a reduction of its provided Caps (there may be logical ones left, which are also provided by another package). There may also be Packages, which only remove Caps without adding new ones (system shrink)!
Versions
This section has become very long and is very crucial regarding its social aspects.
There *have* to be policies to be followed by the package maintainers. I'm currently thinking of version numbers like 1.1.3.4.2 #(1 1 3 4 2) 2.7.3.1.1 #(2 7 3 1 1) , where a change in one of the numbers means:
- incompatible, changed interface semantics, called major package
number: the package is expected to be incompatible with older or newer ones having another number here (at this position);
- downwards compatible, called minor package number: the package has
changed, and their may be new features, but it is expected to work with software written for lower numbers here (but with same 1. number, of course);
- downwards *and* upwards compatible: the package has changed, but
there are *no* new features here (but it may be faster for example), so it is expected to work with software written for higher numbers here (but with same 1., 2. numbers, of course);
- compatible: 'small' fixes without changed functionality in the sense
above (also see 'Classification issues' below);
- no code changes at all: no changed semantics, but e.g. improved class
comments.
After this scheme, a version number a.b.c.d. for a package P corresponds to the Caps 'P', 'P_a', 'P_a.b', 'P_a.b.c', 'P_a.b.c.d', 'P_a.b.c.d.e'. Caps 'P_a.b' - normally - includes *every* Caps 'P_a.x', x < b, since the functionality/interface has just been smaller then. The longer numbers are for expressing conflicts and requirements in a world not being perfect. This means, we finally have defined a relationship between Caps and version numbers! And it should be clear now, why there is this funny plural with the 'Caps' term: it is a normal case for a package-version combo, that it provides a *set* of Capabilities!
I am not comfortable with adding yet another version numbering scheme. It seems simply too complex to me.
SMPackageReleases has today one manually edited field "version" and one automatic version number which can't be edited. I want to express the "level of change" in one release compared to the previous release using categories instead. And I have created a first shot at these categories in SM and made it mandatory to select one of those when you register a new release.
Classification issues
It seems to be difficult to classify fixes (4.) correctly. Fixes definitely change the formal semantics, so one has a good reason to rank them higher and exchange them with 3. for example. A version number change in 3. normally doesn't change the formal semantics (but may introduce new bugs as a change in 4. as well, changing the formal semantics then).
But if a 'small' fix just makes the package working with some seldom used other package being the only one running into the bug, then the classification as 4. makes sense (3. may introduce new bugs affecting many packages). But if the fix removes some serious problems affecting many other packages, it may as well classified as 2., since there is good reason to view this as an important upgrade.
A similar argumentation could be made for changes in the functionality classified as 3.: a piece of code just running much faster without changing the formal semantics, could open up new use cases. This may as well earn a classification in 2..
Currently I think the intelligence of the package maintainers is questioned here, since 'it depends'...
I interpret this to mean that the classification is not a black and white thing - but that it helps a lot to have. I agree.
Note to fixes
An important fix (changed version number 4.) can make the package work the first time with another one. It may make sense - especially if the major number recently has changed - to apply such a fix to an older version, too! This would lead to a new older package with a new number, of course.
Automatically upgrading
In general automatically upgrading should
- apply to all packages increasing their minor or lower ranked version
number,
- *not* apply to packages increasing their major version number.
A harvesting process could be introduced using this technical solution as follows: addditional classifications of packages as stable/unstable/etc. could prefilter the number of packages to be searched for fulfilling dependencies (this classification could also be realised by just putting them into classified - different - package directories).
I think exactly how auto upgrading will work will be heavily parameterized and also ask the user for input. And different people can have different engines - so this isn't crucial stuff to "agree on".
Why major version number 1. at all?
After this versioning policy automatically upgrading would (mostly) work from version to version with a *fixed* first version number. If the major number changes, there are incompatible versions (since we have defined it so!). This could lead to the question, why not to ommit this major number and to force the package maintainer to publish a new package with a different name instead.
Arguments for a major version number increase instead of a name change:
- There are cases for expressing the existence of some functionality at
a very high level independent from the major number: e.g. some package wants to have a package WebBrowserStandard for enforcing to give the user a chance of viewing its generated HTML files (without assuming any API for calling it itself): a newly created name (how to guess it for other package maintainers?) instead of a major number increase would make expressing this dependency impossible.
- Though an interface has changed, the functionality may be very
similar, and the package name should express functionality.
- The package name combined with the major number could be seen as the -
technically and socially - needed new name.
Personally I think the identity of a package should foremost be the identity of the code base evolving. Sure, people can always elect to "fork" a package, by creating a new one - but hopefully in the coming SM that supports branches in the release tree such forks will be even more rare.
Consequences
This is a client POV versioning policy. There may be others, of course: IMHO it's not so important which policy to have, but
*** It's very important to have a policy at all! ***Otherwise a technically working dependency mechanism doesn't work for packages because of social reasons...
Technically this means to define some Caps <-> version number relation with an order of Caps corresponding to an order of version numbers like described above (or to use a similar mechanism).
*** Otherwise the technical solution *cannot* work! ***
But I assume we can still represent the "level of change" using categories instead of version numbers?
Installation of packages
In general there are
- pre-install,
- do-install,
- post-install,
- pre-remove,
- do-remove,
- post-remove,
Scripts (they may be empty). They are triggered by performing an (package install/removal) Action.
Normally the install phase results into adding all provided Caps and the remove phase into the removal of them. Potential conflicts have to be solved *before* installation!
Personally I think we should strive to get away from scripts - they just break. :) Seriously.I would rather try to use a declarative approach as long as possible and then only add hooks for scripts if we find out it is really needed.
Scripts can also seriously trip us up and produce weird results that we will have a hard time tracking down.
Different kind of installs
There are different kind of installs:
- upgrading,
- full installation with a previous removal of an older version (if it
exists).
Yes, currently SM actually differentiates these but only MC has the capability of doing things "differently" when upgrading - and it figures that out on its own today.
Examples (for looking, if the concepts are working) ##################################################
Package WebBrowserStandard
- Provides Caps 'WebBrowser' and more;
- needs Caps 'SocketStuff' and 'RenderingMachine';
- exists in versions 1.1.3.1.7, 2.1.2.3.1, 2.2.1.1.1;
- conflicts with package 'WebBrowserNicerButSlow' and some special
SocketStuff versions.
I note that you have the Caps on the package and not on the releases. This would mean they can't change over time, or rather - the releases can't - at a specific point in time - have different Caps.
I also note you are expressing conflicts on package level, which also suffers from the same problem.
Each package version combo WebBrowserStandard_1.1.3.1.7, WebBrowserStandard_2.1.2.3.1, and WebBrowserStandard_2.2.1.1.1 needs a Transformation describing pre- and post Caps sets.
E.g. for WebBrowser_2.2.1.1.1 there is the following:
a) It provides Caps 'WebBrowserStandard_2.2' being a super set including all Caps provided by WebBrowserStandard_2.1.2.3.1, but *not* all Caps provided by WebBrowser_1.1.3.1.7 (since the major number has changed);
b) It may require Caps ('SocketStuff_3.4', 'RenderingMachine_2'). A package RenderingMachine_2.1 would provide the last requirement.
c) It may conflict
- with package WebBrowserNicerButSlow in general (since it uses the
same sockets); and
- with SocketStuff_3.4.3.7.x (arbitrary x), since the bugfix there had
some very special side effect - unfortunately another installed important package PWorkaround needs exactly one of this versions of SocketStuff: it made a workaround not compatible with a newer version of it, where the side effect has been removed... (world is not perfect).
Note: if the user wants to install WebBrowserNicerButSlow or any other package, all packages have to be checked for conflicts.
How to model c)?
- The conflict with package WebBrowserNicerButSlow is an all over
package conflict and will be modeled as a conflict with Caps ('WebBrowserNicerButSlow'), which includes all Caps ('WebBrowserNicerButSlow_a.b.c.d.e') for arbitrary a, b, c, d, e.
- The conflict with SocketStuff in versions 3.4.3.7.x is *not* an all
over package conflict, it is a package-version conflict. This will be modeled as a conflict with Caps ('SocketStuff_3.4.3.7').
Ok, you differentiate with "all over package" conflicts and "package-version conflicts".
d) It has a Transformation requires: ('SocketStuff_3.4', 'RenderingMachine_2.1') conflicts: ('SocketStuff_3.4.3.7') | V provides: ('WebBrowserStandard', "all over package Caps" 'WebBrowserStandard_2', "major release" 'WebBrowserStandard_2.2', "minor release" 'WebBrowserStandard_2.2.1', 'WebBrowserStandard_2.2.1.1', 'WebBrowserStandard_2.2.1.1.1', 'WebBrowser' "logical Caps"). The last Caps in the result is a logical Caps provided by both packages WebBrowserStandard and WebBrowserNicerButSlow. Such a beast makes sense for expressing dependencies which shouldn't be bounded to a certain package: in this example there may be a point in the system automatically starting the installed WebBrowser, whatever it may be (WebBrowserStandard or WebBrowserNicerButSlow).
The question in case of holding package PWorkaround is, if there is a version SocketStuff_3.a.b.c.x with a >= 4, b, c, x arbitrary, but not ((a = 4) & (b = 3) & (c = 7)), which also gives the input Caps 'SocketStuff_3.4'! Otherwise it wouldn't be possible to install WebBrowserStandard without removing PWorkaround first. So the package-version conflicts are reducing the number of packages capable of providing the needed input caps.
I am sorry Stephan but my brain didn't parse all this. I am fearing the complexity here.
[SNIP of similar description of upgrades that also fried my brain :)]
[SNIP of make tool]
To Lex's example
[SNIP of description of example]
Thinking in capability sets and a versioning policy is needed.
Well... I am not sure it is "needed", but that doesn't mean we can't prepare for it. My summary is at the end.
But testing is nevertheless important: it would be very good to have many automatically runnable tests to classify package combinations as stable/unstable/bleedingEdge/etc.. Since we are faced with a combinatorial explosion here, hints regarding stable/unstable/bleedingEdge/etc. from the package maintainers would be helpful though...
The stable/unstable categories are today mandatory per package release. In other words - when you register a new release today you have to specify maturity level AND compatibility level (giving something similar to your version number scheme).
Relationships with existing tools and concepts ##############################################
SM
If the proposed version policy would be used for classifying packages at SM, the release flag of packages would be replaced or modelled by a corresponding version number, which is much more fine granular.
You are saying "release flag". I assume you mean the "published" flag (though I agree that it could have been called "release" or something). And I don't agree. :) That flag is meant to tell people that even though that release is available on SM it SHOULD NOT BE DEPENDED upon. The maintainer might just be testing it etc. It is a "hey, this release might not even be here - I may delete it at any second"-thing.
And also - the compatibility level categories recently introduced seems very much like your version numbering scheme.
An important question: Who is responsible for maintaining logical packages?
You mean "Caps"?
An - already known - example: packages WebBrowserStandard and WebBrowserNicerButSlow may both generate Caps 'WebBrowser' by an installation. There also may be a logical package WebBrowser which just depends on one of them. But who makes the choice? This is similar to the question, who decides which stuff is worth enough to come into the official distribution.
It would be interesting to enrich SM with a dependency mechanism as described in this paper...
Yes, at least in some regards - see my proposal/summary below.
[SNIPP of some stuff]
Roadmap (loosely, may change) #############################
- Posting this mail and looking for the reactions.
*** If there is enough interest: ***
Yes, sorry for taking a while to respond. It overwhelmed me at first. :)
Creating a Swiki page with this stuff as a start.
Getting a consensus for a versioning policy by
- getting feedback via ML discussions,
- posting an improved draft,
for some iterations. To come to a consensus at this point is *very* important, *independently* from the chosen technical solution!
Yes - or at least we need some form of concensus regarding the structures in SM that can support different approaches.
- Going further with discussions about technical ideas: the concepts
should be good *before* coding. This doesn't mean to discuss forever and not to start with coding after a while...
Personal remarks ################
Don't try to be too perfect: KISS!
Yes. And I am sorry Stephan, but this model is... not really KISS in my book. :) But I like parts of it.
I have to admit that KISS seams to be quite complex in this domain...
Hehe. :)
I'm willingly to invest some more time - e.g. by coding - into Dependencies for Squeak, but there should be a good chance, that the results won't be thrown into the trash can...
We should try to ensure that.
Ok, here is my summary (phew):
1. Caps is an extra "indirection layer" for expressing more abstract dependencies. It should be possible to add "on top" of SM. It is a layer I will not work with myself, because I think it is an overkill. But that doesn't mean we can't make sure that the structures in SM allow for it to be added.
2. Caps in themselves seems to be very easily modelled using SMCategories. The category tree is currently centrally managed by me, in other words - by the Guides. I assume that would be the same for Caps, so it seems to fit.
3. Your version numbering scheme seems overly complex IMHO. I am not sure if the current "Compatibility level"-as-categories-approach fulfills your needs there. Note that the actual sub categories representing each level can easily be changed - I just whipped up a few, see http://map1.squeakfoundation.org/sm/category/cb51b604-bb97-415d-a040-f23 1ecdd5bc7
Well, my head is spinning a bit, but what about this:
I have in my dev image added the concept of resources. A resource is like an "attachment" to another SMObject. This way other tools can attach metadata to for example SMPackages or SMPackageReleases that SM doesn't really know how to handle - but doesn't care about. At this point - since all modifications to the map must be done at the master server - this would only be doable using a remote call to the master server - but we can add those.
I then have a concrete subclass of SMResource called SMPackageReleaseConfiguration. It holds a list of required SMPackageReleases and a status flag. The flag can currently be #working or #failing. #working means that the release we attach the config to "works" with those specific required releases installed. It is thus a tested configuration, as I have described before.
We also already have the mandatory categorizations of releases "Compatibility level" and "Maturity level".
Now - let's say I continue with my "simpler" model that doesn't have Caps and focuses on the SMPackageReleaseConfigurations as the basis for figuring out "what do to". My mechanism is built in a separated SMDependencyEngine. We may even figure out a set of "services" that such an engine should be able to give.
One simply service would be "Ok, given that I want Seaside, Monticello an Shout in this image - what specific installs and upgrades should I perform and in which order?". This is btw the first service I am trying to implement as a spike solution.
So if we agree on a set of such services - or in fact different engines could offer different services and we can just let the engine hook itself into the menues in the package tools and offer "what it has" - then you can build your Caps-based engine on top of SM and use:
- Caps as categories. They would then be optional to use by the package maintainers. - Compatibility level (as exists today, mandatory) - Maturity level (as exists today, mandatory) - SMPackageReleaseConfigurations (as I have described and that I am adding) - SMResource (so that you can optionally add extra metadata to SMObjects inside the map) - And your engine can hook into the SqueakMap Package Loader menus when installed.
Perhaps you are disappointed at me for not buying your solution - I hope not. :) Caps is an interesting approach - even though I think it is overkill. We share the idea of recording the compatibility level of releases - but I don't like encoding it in version numbers. But I desperately want SM to work as a "base" for testing different approaches here, so please - tell me how this sounds and what you think/need to consider moving forward with your idea.
regards, Göran
Hi Göran and all!
Göran, thanks for your constructive reply.
After reading it once, I have the impression, that this could lead to an interesting debate *improving* the concepts.
<Some time later> I'm a little bit tired now... ;-)
I've sitted one night to write this answer and many things have become clearer while thinking about and writing the answers followed by iterated thinking about previously treated parts. I've been faced with misunderstandings and hopefully my thoughts about your thoughts are coming close to reality now... So I suggest to read this to the end before hitting the reply button and starting writing, many things should become clearer while going this long way and if gotten the whole picture...
Göran, please don't feel forced to reply fast, I really understand if this takes a while...
But now let's go!
At first I want to clearify some definitions to avoid confusion, which has arosen on both sides, I think.
I've used the term Package (uppercase 'P') *always* in the narrower sense of 'one package-version combo', e.g. Package WebBrowser_2.1.3.4.1. There is admittedly at least one exception (the headerline 'Package WebBrowser' below obviously denotes a bunch of versions, but it is just 'packages' written as section headerline with uppercase first letter).
And I've used the term 'package' context dependent: sometimes as Package, but also sometimes more general, e.g. package WebBrowser. I have found this more general use in *opposite* to your definition, which seem to match the definition of DEPS Packages above, as I see it now (at least these interpretations have led to a consistent picture in my head).
I think we have an overloading of the term 'package', which leads to confusion.
Is 'package' for you a shortcut for a package-version combo, which would be the same as a DEPS Package above? Do you mean with 'release' one of many packages (package-version combo), stating that something important has changed in the code, of some kind of "Don't know how to avoid the word 'pack...'" doing something (e.g. a WebBrowser "Don't know...")?
Another problem: I've also been confused by the term 'release': I would expect to have only releases - tagged stable versions in a version control sense - at SM, but you seem to have another definition/policy/whatever there...
I hope this helps for reading the rest (there is at least one highlight waiting for you :-) ):
goran.krampe@bluefish.se wrote:
Hi Stephan and all!
I got a bit tired of writing long postings, but since I definitely don't want to loose the opportunity to cooperate with Stephan here I will take the time to reply to this one know.
Before I start - Stephan - don't be offended if I don't agree with you now. :)
No problem.
And I will try to snip parts that I have no comment on.
Good (I have asked for it).
Stephan Rudlof sr@evolgo.de wrote: [SNIP]
Technical Terms ###############
Caps
There are Capabilities (Caps) each describing the capability to do something. Caps mostly depend on other Caps, which are prerequisites of them. In the following Caps means Capabilities or a union of them.
This "Caps" layer is an extra abstraction layer that I am not sure about. I understand the theory behind it - but it feels a bit on the overly complex side, at least if it is the primary mechanism. If it was an additional abstraction that one could select to use or not, then it would be another matter.
But I will read on first... :)
Transformations
Transformations describe how it is possible to change a set of Caps: they may - virtually at first - add or remove Caps or both. They transform one set of Caps into another one. To do their work they perform Actions.
Inputs are I.1. requires: required Caps, and I.2. conflicts: conflicting Caps, outputs are O.1. provides: newly generated Caps, and O.2. deletes: deleted Caps. Properties:
- provides are const sets, but
- deletes are configuration dependent.
Outputs are expressed as
- set of (named) Caps.
Inputs are expressed as
- set of (named) Caps (as for outputs), or
- as a Block
- taking a Caps set as an argument, returning
- true, if the requirement/conflict has been fullfilled,
- false, otherwise.
Hmmm, ok - at first look this seems again pretty... advanced. Or complex.
To Blocks as input
It is computational expensive to use Blocks as input, since in principle all existing Caps and those somehow generatable by whatever Transformation combination (!) have to be checked.
An - important, I think - improvement for the case of expressing conflicts may be to divide the Blocks into two classes (later there may be more):
- Blocks just checking conflicts with Caps generated additionally by
the packages generating the requirement Caps, under the precondition, that the requirement Caps are given as set. 2. Other Blocks.
See section "To Lex's example" below for a motivation of Blocks as arguments here and the suggested improvement.
Ok, this part isn't at all clear to me at this point. But it is only a reflection so...
Boolean logic
Require sets model a logical AND of requirements. Logical ORs - and therefrom Boolean logic - can be modeled via additional Transformations with newly introduced Caps names. This is far more efficient as trying to do this in one Transformation using the Blocks mechanism!
Ouch. Ok, my first reaction is "NO!". :) I know it is tempting to add the capability to use boolean logic to express dependencies/scenarios or whatever - and Debian uses it - but personally I have elected to NOT go there because the complexity seems to me to be unwarranted.
If I've understood you correctly below, you also have AND logic (multiple requires) in your logic. A logical Caps just extends this to also have an OR.
How would you model something like WebBrowserUsingApp requires: (WebBrowserStandard | WebBrowserNicerButSlow) ; without some kind of Boolean logic?
With Caps there are two ways to go in this example.
The first one assumes there is no logical Caps 'WebBrowser'. Then there is a need to introduce two new - very simple - Transformations
requires: ('WebBrowserStandard') | V provides: ('WebBrowser').
requires: ('WebBrowserNicerButSlow') | V provides: ('WebBrowser').
and to require 'WebBrowser' in the 'WebBrowserUsingApp' generating Transformation.
The second alternative is chosen below: there the idea of having a logical 'WebBrowser' has been at 'package providing/deleting Transformation' creation time of WebBrowserStandard and WebBrowserNicerButSlow, so this Caps 'WebBrowser' has just been added to their provides then. Note: Puh, what a huge word combo is "at 'package providing/deleting Transformation' creation time" ! But I've wanted to be as precise as possible. I think we need some definitions, but not now...
The good think here is, that it is possible to *add* such a logical choice *afterwards*, if it hasn't been done at writing the Transformations for one of the Browser packages, without the need to change them! So later there can be better and better Transformation rules, with different originators: there needn't be one central instance (without such a mechanism you had to change the orginal Transformations in direction of the second alternative.
Note: This example doesn't forbid to ask the user for a decision if the dependeny resolver comes to this point. But this is an UI issue.
To the complexity issue: What is complex here?
I don't see any problems into traversing a graph of nested AND/OR connected requires and trying the alternatives if there is an OR (the AND exists in every non-trivial dependency logic, I assume).
Am I missing something?
But ok, shutting up and reading on... :)
Outlook to packages
A package removal can be modeled by transforming the all over package Caps as requires into deletes of this and all other package related Caps.
Don't understand, but not important at this point I guess.
How to model dynamic provides?
The restriction 'provides are const sets' eases some things, but it is not a limitation AFAICS.
Example: Installing a package WebBrowser may result into more Caps if some other package SuperFonts is there. This can easily be modeled by a *logical* (no Action) package Transformation describing the all over functionality of the packages combo, with
- requires: Caps ('WebBrowser', 'SuperFonts'), and
- provides: something like ('WebBrowserWithSuperFonts').
Ok, at this point I am thinking - perhaps this Caps layer should be something on top of SM? This may mean that your model could be a layer on top of mine. Just a feeling at this point.
Note
It's questionable, if naming both set and Block inputs the same is a good idea, but I want to stop for now...
Actions
Actions are commands leading to a change of the (set of) Caps in the system. If triggered, they add and/or remove one or more Caps in reality.
Algorithms ############
Before there is no good model of the domain, it is not urgent to think about algorithms. I don't fear not to find some...
A few things I'm currently expecting to be faced with: Easy:
- depth first traversing of dependency graphs,
- back tracking with detection of cycles.
Not so easy:
- combinatorial explosion, if Blocks as inputs for Transformation
requires and conflicts come into play (but section 'Transformations->To Blocks as input' contains a first idea for embanking it).
Yes, the reason for me starting to write code was that I wanted to explore at least some of the algorithms involved.
Application to packages #######################
Packages
A Package provides Caps: these are representing some things to be made with or by the Package (interfaces, commands). There may be more than one Caps per Package!
Since this would be a thing changing over time I would presume that actually a package *release* provides Caps?
After my definitions: - *Each* DEPS Package provides a *set* of Caps. - Some sets of package-version combos (or Packages) are sharing the same Caps (one or more). - A set of package-version combos belonging to the same e.g. WebBrowser general package shares one or more of the provided Caps! At least after the proposed versioning number scheme below...
So a 'package *release*' - as I've understood you - has the following properties in DEPS notation: - it provides a set of Caps, - it should have a change in one of the higher ranked version numbers (to express the semantically important code change), - it provides a Caps (in the set of its provided Caps) shared by a set of Packages (all are providing it) containing this release and all sub-releases (this could be used to express a dependency or conflict to a set of Packages at release compatibility level).
I think this will become clearer if you read further to more versioning number examples and - neat outlook - how to map them to categories and vice-versa!
A Package needs Caps provided from other packages (requires).
Same thing here. In my model this extra "redirection layer" is not present - there a release has tested configurations and each configuration more or less lists the other package *releases* that are needed for the release to work.
I see a problem here with your approach: you are faced with a combinatorial explosion. One package release requiring say 4 other packages each existing in say 4 releases leads to 4 ^ 4 = 256 configurations...
Let's assume, there are 10 tested configurations. Then you have to switch to one of the 246 untested ones if there is only one other general package requiring just one of the 4 requires in another release (assumed that different versions cannot coexist).
To help me understand: Why aren't you seeing a problem here?
If I've understood you correctly, you would try to come through this via the compatibility level categories combined with some rules for controlled deviation from the tested configurations, but currently I have serious stomaches here.
Which made me think perhaps this capability layer can be "on top" of my model.
More to this point below.
A Package providing some Caps may conflict with another Package (providing other and/or the same Caps).
To be able to express these conflicts at Caps level it is necessary that all packages are mutually different in at least one Caps! But this is easy to accomplish, if there is just one Caps named so, that it contains the - unique - name of them.
Note that a Cap could easily be modelled using SMCategories. That is the point with SMCategory - it is a unique canonicalized "tag" that can be attached to SMObjects.
I am not sure what you are saying though in the last part. Are you saying that two different packages (releases) can not provide the exact same Caps?
Exactly. And they don't after the proposed versioning scheme.
Some explanation, though: here Caps in the sense of union of Caps is meant. E.g. general package Foo in version 4.4.3.2.3 provides standard Caps (via a corresponding Transformation) ('Foo', 'Foo_4', 'Foo_4.4', 'Foo_4.4.3', 'Foo_4.4.3.2', 'Foo_4.4.3.2.3') ; another minimal changed version 4.4.3.2.4 of the same general package provides standard Caps ('Foo', 'Foo_4', 'Foo_4.4', 'Foo_4.4.3', 'Foo_4.4.3.2', 'Foo_4.4.3.2.4') . In this example both versions share all, but the last Caps.
Note: this representation is made from/for a semantic point of view; in an implementation one #Foo_4.4.3.2.4 may be sufficient to represent all standard Caps (but there may be more).
Why not?
To be able to differentiate between two different package-version combos at the Caps level. This is needed to be able to express a conflict with just this special seldom used one version of package Foo (e.g. expressed as conflict with Caps 'Foo_4.4.3.2.3').
A conflicting Package has to be removed (this can be difficult) before installing a new one, leading into a reduction of its provided Caps (there may be logical ones left, which are also provided by another package). There may also be Packages, which only remove Caps without adding new ones (system shrink)!
Versions
This section has become very long and is very crucial regarding its social aspects.
There *have* to be policies to be followed by the package maintainers. I'm currently thinking of version numbers like 1.1.3.4.2 #(1 1 3 4 2) 2.7.3.1.1 #(2 7 3 1 1) , where a change in one of the numbers means:
- incompatible, changed interface semantics, called major package
number: the package is expected to be incompatible with older or newer ones having another number here (at this position);
- downwards compatible, called minor package number: the package has
changed, and their may be new features, but it is expected to work with software written for lower numbers here (but with same 1. number, of course);
- downwards *and* upwards compatible: the package has changed, but
there are *no* new features here (but it may be faster for example), so it is expected to work with software written for higher numbers here (but with same 1., 2. numbers, of course);
- compatible: 'small' fixes without changed functionality in the sense
above (also see 'Classification issues' below);
- no code changes at all: no changed semantics, but e.g. improved class
comments.
After this scheme, a version number a.b.c.d. for a package P corresponds to the Caps 'P', 'P_a', 'P_a.b', 'P_a.b.c', 'P_a.b.c.d', 'P_a.b.c.d.e'. Caps 'P_a.b' - normally - includes *every* Caps 'P_a.x', x < b, since the functionality/interface has just been smaller then. The longer numbers are for expressing conflicts and requirements in a world not being perfect. This means, we finally have defined a relationship between Caps and version numbers! And it should be clear now, why there is this funny plural with the 'Caps' term: it is a normal case for a package-version combo, that it provides a *set* of Capabilities!
I am not comfortable with adding yet another version numbering scheme. It seems simply too complex to me.
I have more feared, that it could be too simple ;-) As I've written in another Mail to Stan: "The granularity of the proposed scheme is fine, but better too fine as too simple: nobody is forced to publish versions only changing in lower ranked version numbers, but it is possible. And it is also possible to be very fine granular in expressing conflicts with other package versions: from a singular version to larger and larger sets of versions."
I imagine one simple dialog asking a package release publisher five or less questions for computing a new version number (for all increase rules applies, don't change higher ranked version numbers, reset lower ranked ones (similar to numbers ;-) )):
1. Do you expect, that most code running with the previous version, does *not* work anymore with this new version? yes -> increase 1., finished; no -> continue...
2. Do you expect, that most code running with the previous version, continues to work with this new version? yes -> increase 2., finished; no -> continue...
3. Do you expect, that most code to be written for running with this new version runs with the previous version, too? yes -> increase 3., finished; no -> continue...
4. Do you expect, that 3. applies *and* have you made a small fix without adding new functionality? yes -> increase 4., finished; no -> continue...
5. Do you have *not* changed any code at all (just comments, etc.), so that's *impossible* that other code breaks? yes -> increase 5., finished.
Of course with 'version' is meant a published/public version *** thought to be used by others *** here, so these questions don't apply to 'private' versions (probably they shouldn't go into the dependency system at all, but manually installed).
It should be possible to answer these maximal 5 questions for a package maintainer (beeing the target group here), otherwise I wouldn't like to have his/her package... ;-)
SMPackageReleases has today one manually edited field "version" and one automatic version number which can't be edited. I want to express the "level of change" in one release compared to the previous release using categories instead. And I have created a first shot at these categories in SM and made it mandatory to select one of those when you register a new release.
If you'd choose the categories along the questions above, the DEPS version numbers could be computed automatically (coming again as highlight below)...
Some prospects for convergation here :-)
Classification issues
It seems to be difficult to classify fixes (4.) correctly. Fixes definitely change the formal semantics, so one has a good reason to rank them higher and exchange them with 3. for example. A version number change in 3. normally doesn't change the formal semantics (but may introduce new bugs as a change in 4. as well, changing the formal semantics then).
But if a 'small' fix just makes the package working with some seldom used other package being the only one running into the bug, then the classification as 4. makes sense (3. may introduce new bugs affecting many packages). But if the fix removes some serious problems affecting many other packages, it may as well classified as 2., since there is good reason to view this as an important upgrade.
A similar argumentation could be made for changes in the functionality classified as 3.: a piece of code just running much faster without changing the formal semantics, could open up new use cases. This may as well earn a classification in 2..
Currently I think the intelligence of the package maintainers is questioned here, since 'it depends'...
I interpret this to mean that the classification is not a black and white thing
Yes.
- but that it helps a lot to have.
I agree.
I have to think here. Without this classification there would just be standard Caps like e.g. for general package Foo in version 4.4.3.2.3: ('Foo', 'Foo_4_4_3_2_3') ('_' instead of '.' for not thinking they are made from real DEPS versions), and you wouldn't have any semantically meaningful relationship of the latter (version) Caps 'Foo_4_4_3_2_3' to Caps 'Foo_4_4_3_2_4' of the next version. So 'Foo_4_4_3_2_4' is equally 'far' away from 'Foo_4_4_3_2_3' than e.g. 'Foo_9_9_9_9_9'.
Automatically upgrading wouldn't work; so classification is *needed* here. The classification may be wrong, but this can be repaired via more/bigger Transformations and/or prefiltering (e.g. by disabling packages with a compatibility classification too optimistic (too pessimistic doesn't break anything)) of packages to be considered for Transformations at all.
Note to fixes
An important fix (changed version number 4.) can make the package work the first time with another one. It may make sense - especially if the major number recently has changed - to apply such a fix to an older version, too! This would lead to a new older package with a new number, of course.
Automatically upgrading
In general automatically upgrading should
- apply to all packages increasing their minor or lower ranked version
number,
- *not* apply to packages increasing their major version number.
A harvesting process could be introduced using this technical solution as follows: addditional classifications of packages as stable/unstable/etc. could prefilter the number of packages to be searched for fulfilling dependencies (this classification could also be realised by just putting them into classified - different - package directories).
I think exactly how auto upgrading will work will be heavily parameterized
Agreed.
and also ask the user for input.
Agreed, but with limitation: An end user just using Squeak only wants to have a successful installation with as less questions as possible (and the end user often wouldn't know how to answer).
And different people can have different engines
Yes. But there will be some default.
- so this isn't crucial
stuff to "agree on".
Please don't be so fast here: - as better the classifications, as better the automatically upgrading; - as worse the classifications, as worse the automatically upgrading!
This won't be changed by another parametrization or engine. Or did I miss a point here?
Why major version number 1. at all?
After this versioning policy automatically upgrading would (mostly) work from version to version with a *fixed* first version number. If the major number changes, there are incompatible versions (since we have defined it so!). This could lead to the question, why not to ommit this major number and to force the package maintainer to publish a new package with a different name instead.
Arguments for a major version number increase instead of a name change:
- There are cases for expressing the existence of some functionality at
a very high level independent from the major number: e.g. some package wants to have a package WebBrowserStandard for enforcing to give the user a chance of viewing its generated HTML files (without assuming any API for calling it itself): a newly created name (how to guess it for other package maintainers?) instead of a major number increase would make expressing this dependency impossible.
- Though an interface has changed, the functionality may be very
similar, and the package name should express functionality.
- The package name combined with the major number could be seen as the -
technically and socially - needed new name.
Personally I think the identity of a package should foremost be the identity of the code base evolving.
Sure, people can always elect to "fork" a package, by creating a new one
- but hopefully in the coming SM that supports branches in the release
tree such forks will be even more rare.
Increasing 1. can be considered as starting a fork (after the proposed DEPS versioning scheme).
While thinking about this there has come the following question into my mind: what is the difference between a versioning scheme needed for a package dependency system with *automatically* upgrading, and a versioning scheme needed for a version control system?
The main difference - as I currently see it - is, that - for the package dependency system, versioning should reflect the compatibility *categorization/classification*, which is a pure semantical thing; and - for the version control system, versioning just says, how far two versions are *technically* away (e.g. by applying patches), which as itself says *nothing* about the semantical changes.
Consequences
This is a client POV versioning policy. There may be others, of course: IMHO it's not so important which policy to have, but
*** It's very important to have a policy at all! ***
Otherwise a technically working dependency mechanism doesn't work for packages because of social reasons...
Technically this means to define some Caps <-> version number relation with an order of Caps corresponding to an order of version numbers like described above (or to use a similar mechanism).
*** Otherwise the technical solution *cannot* work! ***
But I assume we can still represent the "level of change" using categories instead of version numbers?
Here comes the promised *highlight* a little bit more detailed:
If each package-version combo gets categories corresponding to answers to the questions to package maintainers as written above: then we even have a *mapping* from categories to version numbers (incrementally computed) and vice versa (direct mapping).
Isn't that nice?
Installation of packages
In general there are
- pre-install,
- do-install,
- post-install,
- pre-remove,
- do-remove,
- post-remove,
Scripts (they may be empty). They are triggered by performing an (package install/removal) Action.
Normally the install phase results into adding all provided Caps and the remove phase into the removal of them. Potential conflicts have to be solved *before* installation!
Personally I think we should strive to get away from scripts - they just break. :)
Seriously.I would rather try to use a declarative approach as long as possible and then only add hooks for scripts if we find out it is really needed.
I didn't want to give a definition for 'script' here; this part has mainly been written down to show, where I see some reasonable hooks, whatever 'scripts' should be. Better had been to replace "Scripts (they may be empty)." by "Scripts (they often are empty)."
Scripts can also seriously trip us up and produce weird results that we will have a hard time tracking down.
Yes, of course.
Different kind of installs
There are different kind of installs:
- upgrading,
- full installation with a previous removal of an older version (if it
exists).
Yes, currently SM actually differentiates these but only MC has the capability of doing things "differently" when upgrading - and it figures that out on its own today.
Examples (for looking, if the concepts are working) ##################################################
Package WebBrowserStandard
'Package': normally I've meant the package-version combo, here the more general 'package' has just been written as headerline...
- Provides Caps 'WebBrowser' and more;
- needs Caps 'SocketStuff' and 'RenderingMachine';
- exists in versions 1.1.3.1.7, 2.1.2.3.1, 2.2.1.1.1;
- conflicts with package 'WebBrowserNicerButSlow' and some special
SocketStuff versions.
I note that you have the Caps on the package and not on the releases. This would mean they can't change over time, or rather - the releases can't - at a specific point in time - have different Caps.
They can! Via changed and/or newly added Transformations. Caps besides the standard Caps (since they stay forever) could be changed arbitrarily.
I also note you are expressing conflicts on package level, which also suffers from the same problem.
For me one package-version combo (hopefully the same as exactly one installable 'package') is the finest granular entity. It has a *set* of standard Caps; e.g. (not in ST notation)
('Foo', 'Foo_4', 'Foo_4.4', 'Foo_4.4.3', 'Foo_4.4.3.2', 'Foo_4.4.3.2.3')
You can express conflicts to just this one package Foo_4.4.3.2.3 via a conflict with (one of the standard) Caps 'Foo_4.4.3.2.3' only provided by this package. But it is also possible to express a conflict to some set of packages, e.g. by a stating a conflict with Caps 'Foo_4.4' many versions are meant, exactly those all providing this Caps (e.g. packages Foo_4.4.7.3.3, Foo_4.4.7.3.3).
Each package version combo WebBrowserStandard_1.1.3.1.7, WebBrowserStandard_2.1.2.3.1, and WebBrowserStandard_2.2.1.1.1 needs a Transformation describing pre- and post Caps sets.
E.g. for WebBrowser_2.2.1.1.1 there is the following:
a) It provides Caps 'WebBrowserStandard_2.2' being a super set including all Caps provided by WebBrowserStandard_2.1.2.3.1, but *not* all Caps provided by WebBrowser_1.1.3.1.7 (since the major number has changed);
b) It may require Caps ('SocketStuff_3.4', 'RenderingMachine_2'). A package RenderingMachine_2.1 would provide the last requirement.
c) It may conflict
- with package WebBrowserNicerButSlow in general (since it uses the
same sockets); and
- with SocketStuff_3.4.3.7.x (arbitrary x), since the bugfix there had
some very special side effect - unfortunately another installed important package PWorkaround needs exactly one of this versions of SocketStuff: it made a workaround not compatible with a newer version of it, where the side effect has been removed... (world is not perfect).
Note: if the user wants to install WebBrowserNicerButSlow or any other package, all packages have to be checked for conflicts.
How to model c)?
- The conflict with package WebBrowserNicerButSlow is an all over
package conflict and will be modeled as a conflict with Caps ('WebBrowserNicerButSlow'), which includes all Caps ('WebBrowserNicerButSlow_a.b.c.d.e') for arbitrary a, b, c, d, e.
- The conflict with SocketStuff in versions 3.4.3.7.x is *not* an all
over package conflict, it is a package-version conflict. This will be modeled as a conflict with Caps ('SocketStuff_3.4.3.7').
Ok, you differentiate with "all over package" conflicts and "package-version conflicts".
"All over package" also denoted "general package" or in this mail (#$% confusion).
d) It has a Transformation requires: ('SocketStuff_3.4', 'RenderingMachine_2.1') conflicts: ('SocketStuff_3.4.3.7') | V provides: ('WebBrowserStandard', "all over package Caps" 'WebBrowserStandard_2', "major release" 'WebBrowserStandard_2.2', "minor release" 'WebBrowserStandard_2.2.1', 'WebBrowserStandard_2.2.1.1', 'WebBrowserStandard_2.2.1.1.1', 'WebBrowser' "logical Caps"). The last Caps in the result is a logical Caps provided by both packages WebBrowserStandard and WebBrowserNicerButSlow. Such a beast makes sense for expressing dependencies which shouldn't be bounded to a certain package: in this example there may be a point in the system automatically starting the installed WebBrowser, whatever it may be (WebBrowserStandard or WebBrowserNicerButSlow).
The question in case of holding package PWorkaround is, if there is a version SocketStuff_3.a.b.c.x with a >= 4, b, c, x arbitrary, but not ((a = 4) & (b = 3) & (c = 7)), which also gives the input Caps 'SocketStuff_3.4'! Otherwise it wouldn't be possible to install WebBrowserStandard without removing PWorkaround first. So the package-version conflicts are reducing the number of packages capable of providing the needed input caps.
I am sorry Stephan but my brain didn't parse all this.
I hope now it has been easier.
I am fearing the complexity here.
It is easier as it seems, I think.
[SNIP of similar description of upgrades that also fried my brain :)]
[SNIP of make tool]
I've made some efforts to give examples: <rant> If you don't give examples, people want to have them. But if you give them, what they want, they don't like them... </rant>
To Lex's example
[SNIP of description of example]
Thinking in capability sets and a versioning policy is needed.
Well... I am not sure it is "needed", but that doesn't mean we can't prepare for it. My summary is at the end.
I'm holding my opinion of needing a good versioning scheme for a good automatically upgrading mechanism. This versioning scheme may correspond to categories as I've described above. The capability set thinking just allows to group the versions as needed for automatically upgrading.
But testing is nevertheless important: it would be very good to have many automatically runnable tests to classify package combinations as stable/unstable/bleedingEdge/etc.. Since we are faced with a combinatorial explosion here, hints regarding stable/unstable/bleedingEdge/etc. from the package maintainers would be helpful though...
The stable/unstable categories are today mandatory per package release. In other words - when you register a new release today you have to specify maturity level AND compatibility level (giving something similar to your version number scheme).
These categories are good for prefiltering of packages to be visible (or not) by the dependency resolver! In my humble thinking... ;-)
Relationships with existing tools and concepts ##############################################
SM
If the proposed version policy would be used for classifying packages at SM, the release flag of packages would be replaced or modelled by a corresponding version number, which is much more fine granular.
You are saying "release flag". I assume you mean the "published" flag (though I agree that it could have been called "release" or something). And I don't agree. :) That flag is meant to tell people that even though that release is available on SM it SHOULD NOT BE DEPENDED upon. The maintainer might just be testing it etc. It is a "hey, this release might not even be here - I may delete it at any second"-thing.
OK, bought your point. Another good prefiltering category.
And also - the compatibility level categories recently introduced seems very much like your version numbering scheme.
They are ;-)
An important question: Who is responsible for maintaining logical packages?
You mean "Caps"?
I mean additional Transformations providing logical Caps in addition to the standard Caps.
An - already known - example: packages WebBrowserStandard and WebBrowserNicerButSlow may both generate Caps 'WebBrowser' by an installation. There also may be a logical package WebBrowser which just depends on one of them. But who makes the choice? This is similar to the question, who decides which stuff is worth enough to come into the official distribution.
It would be interesting to enrich SM with a dependency mechanism as described in this paper...
Yes, at least in some regards - see my proposal/summary below.
[SNIPP of some stuff]
Roadmap (loosely, may change) #############################
- Posting this mail and looking for the reactions.
*** If there is enough interest: ***
Yes, sorry for taking a while to respond. It overwhelmed me at first. :)
I feel with you and experience hard work feelings while writing this reply, too... But I think it is worth it!
Creating a Swiki page with this stuff as a start.
Getting a consensus for a versioning policy by
- getting feedback via ML discussions,
- posting an improved draft,
for some iterations. To come to a consensus at this point is *very* important, *independently* from the chosen technical solution!
Yes - or at least we need some form of concensus regarding the structures in SM that can support different approaches.
This sounds promising. But to get a good automatically upgrading... you know my point.
One more: automatically upgrading also comprises installing a new general package needing some upgraded version and at the same time staying *compatible* with another installed package written for the not upgraded version.
- Going further with discussions about technical ideas: the concepts
should be good *before* coding. This doesn't mean to discuss forever and not to start with coding after a while...
Personal remarks ################
Don't try to be too perfect: KISS!
Yes. And I am sorry Stephan, but this model is... not really KISS in my book. :) But I like parts of it.
I have to admit that KISS seams to be quite complex in this domain...
Hehe. :)
Since the domain is complex, if you want to have a *good* result (automatically...) :)
I'm willingly to invest some more time - e.g. by coding - into Dependencies for Squeak, but there should be a good chance, that the results won't be thrown into the trash can...
We should try to ensure that.
Ok, here is my summary (phew):
- Caps is an extra "indirection layer" for expressing more abstract
dependencies. It should be possible to add "on top" of SM. It is a layer I will not work with myself, because I think it is an overkill. But that doesn't mean we can't make sure that the structures in SM allow for it to be added.
My suggestion here: please try to find arguments why the "overkill" isn't needed for a good automatically..., and how to get it with more modest concepts. Just trying to find them helps in understanding the domain better!
- Caps in themselves seems to be very easily modelled using
SMCategories. The category tree is currently centrally managed by me, in other words - by the Guides. I assume that would be the same for Caps, so it seems to fit.
This is interesting, since it could be a way for trying both approaches in parallel: just one category for the DEPS version number (or more along the mapping highlight described above, but this could be "overkill") would be sufficient to realize a DEPS prototype, I think.
The water in the wine: the install/deinstall hooks for general packages should have some well defined properties, I haven't tried to describe in greater detail so far. But this applies - again - to whatever *good* automatically ...
- Your version numbering scheme seems overly complex IMHO. I am not
sure if the current "Compatibility level"-as-categories-approach fulfills your needs there. Note that the actual sub categories representing each level can easily be changed - I just whipped up a few, see http://map1.squeakfoundation.org/sm/category/cb51b604-bb97-415d-a040-f23 1ecdd5bc7
Just taking a look... I'm surprised (seeing this page the first time, I think): this goes *** very close *** in direction of DEPS versioning scheme! OK, the representation varies, but this is no problem, if my mapping idea above holds.
Well, my head is spinning a bit, but what about this:
I have in my dev image added the concept of resources. A resource is like an "attachment" to another SMObject. This way other tools can attach metadata to for example SMPackages or SMPackageReleases that SM doesn't really know how to handle - but doesn't care about. At this point - since all modifications to the map must be done at the master server - this would only be doable using a remote call to the master server - but we can add those.
I then have a concrete subclass of SMResource called SMPackageReleaseConfiguration. It holds a list of required SMPackageReleases and a status flag. The flag can currently be #working or #failing. #working means that the release we attach the config to "works" with those specific required releases installed. It is thus a tested configuration, as I have described before.
Is a SMPackageRelease one specific version (marked as release or so) or a set of possible versions each fulfilling the same definition of belonging to the same release? I'm somewhat in the fog here.
We also already have the mandatory categorizations of releases "Compatibility level" and "Maturity level".
Now - let's say I continue with my "simpler" model that doesn't have Caps and focuses on the SMPackageReleaseConfigurations as the basis for figuring out "what do to". My mechanism is built in a separated SMDependencyEngine. We may even figure out a set of "services" that such an engine should be able to give.
One simply service would be "Ok, given that I want Seaside, Monticello an Shout in this image - what specific installs and upgrades should I perform and in which order?". This is btw the first service I am trying to implement as a spike solution.
So if we agree on a set of such services - or in fact different engines could offer different services and we can just let the engine hook itself into the menues in the package tools and offer "what it has" - then you can build your Caps-based engine on top of SM and use:
- Caps as categories. They would then be optional to use by the package
maintainers.
Should work. Just needed for additional Caps to the standard ones.
- Compatibility level (as exists today, mandatory)
Should work.
- Maturity level (as exists today, mandatory)
Good for prefiltering.
- SMPackageReleaseConfigurations (as I have described and that I am
adding)
I think I wouldn't need them, but I could need the anti-configurations you have recently mentioned, for expressing conflicts. Perhaps.
- SMResource (so that you can optionally add extra metadata to
SMObjects inside the map)
Always good to have this possibility: is there a 1-to-1 relationship between an SMRessoure and a package-version combo (installable entity)?
- And your engine can hook into the SqueakMap Package Loader menus when
installed.
This sounds promising!
Perhaps you are disappointed at me for not buying your solution - I hope not. :)
I'm not disappointed, since I - think this mail will help to get the different concepts clearer; - haven't seen any serious flaws in my concepts so far (but they may come later, of course); since we - have found a mapping between DEPS version numbers and categories; and you - seem to be open minded and - don't seem to fear a potential competitor (this is great!).
Caps is an interesting approach - even though I think it is overkill.
I don't think so ;-)
We share the idea of recording the compatibility level of releases -
Yes.
but I don't like encoding it in version numbers.
It's just an encoding: what do you think about the 'mapping highlight'?
But I desperately want SM to work as a "base" for testing different approaches here, so please - tell me how this sounds
This sounds good!
and what you think/need to consider moving forward with your idea.
I'd like to continue this discussion until I'm sure, - that you have understood my points, - that I have understood your points; with the goal to better model the domain for getting a good au... ;-)
Greetings Stephan
regards, Göran
Hey Stephan, I enjoyed reading this. By all means continue to explore in this direction and to refine your essay, so that people new to the topic can get going quickly!
Here are a few quick thoughts on it.
First, if I'm not mistaken, your capabilities model seems to boil down, for each package, to this form:
(cap1 AND cap2 AND cap3) IMPLIES (cap4 AND cap5) (cap6 AND cap7) IMPLIES (cap8 AND cap9 AND cap10) Is this correct? You'd ask the installer tool to find a way to get you cap10, and then it would trace backwards through all the packages to know about to see if it can satisfy this.
Do you have any concrete examples where these transformers would be useful, Stephan? It would help in thinking about them.
One aspect of baking in dependencies into version numbers is that people may well end up not wanting to use the dependency-accurate version numbers as their normal version numbers. If am working on Chuck II Son of Chuck(y), then I just want version numbers that go 2.0, 2.1, 2.2, etc., but those will not capture the dependency information accurately. This makes me wary of capturing *accurate* dependency information in the versions, though it may still be that the *intent* of the dependencies can still be encoded in them.
Along these lines, note that a conflict can be detected after the version number has already been decided. What happens in that case? If the version number only captures intent then there is no problem. If it is supposed to be accurate, then that seems impossible.
- The package name combined with the major number could
be seen as the - technically and socially - needed new name.
You beat me to it. For packaging purposes, it gives you a small simplification if Chuck1 and Chuck2 are simply different packages if they are in fact incompatible. Additionally, putting them in separate packages means that you can arrange for *both* of them to be installed simultaneously....
('WebBrowserStandard_2.2', "minor release" 'WebBrowserStandard_2.2.1', 'WebBrowserStandard_2.2.1.1', 'WebBrowserStandard_2.2.1.1.1')."
Keep in mind, everyone, that accurate dependencies require extra work to happen somewhere. I hope stuff like the above does not need to be maintained by humans! And anyway, it makes me wonder in a lot of cases if it is not simpler to *fix* a conflict between two packages and upload a new version of the package, as opposed to trying to accurately report that Chuck 1.28 is incompatible with RB 2.11. Note that we have to deal with bugs anyway, and that an incompatibility can often be considered simply a bug.
But here again: we *need* a versioning policy, which people are following!
Yes, that sounds right to me. Bugfixes happen, and we should have a system that can deal rationally with them. Versioning policies would let you easily (hopefully) express the claims about compatibility that people often have mentally.
. Since we are faced with a
combinatorial explosion here, hints regarding stable/unstable/bleedingEdge/etc. from the package maintainers would be helpful though...
I suggest we solve this through the idea of package universes. You simply live in a bleeding edge universe or a stable one. I don't see how you can combine a bleeding edge package and a stable package and get something meaningful. The result is not stable, for sure.
There's no explosion in practice. The simple "Stable" versus "Unstable" goes a long way.
There also may be a logical package WebBrowser which just depends on one of them. But who makes the choice? This is similar to the question, who decides which stuff is worth enough to come into the official distribution.
Exactly!! There is no one answer to this. A package universe can be viewed as *one* answer to the question "what packages exist?"
It would be nice to have a detection of class extension conflicts.
This sounds like a job for the package loader and unloader, and if I'm not mistaken Monticello will do this already.
Don't try to be too perfect: KISS! I have to admit that KISS seams to be quite complex in this domain...
I am not convinced. If we restrict attention to individual package universes, then simple unversioned dependencies seem to be sufficient. It is worth trying to improve on them, but I don't see why we *have* to improve on them. How much talking *about* our packages do we really want to be doing, as opposed to actually improving the packages themselves? :)
-Lex
PS -- There is a non-trivial UVersion in the Universes prototype I posted, but it is way shy of the kind of stuff you are talking about. I have made -- KISS remember :) -- the assumption that incompatible packages have different names, and that newer versions of packages are prefered even if someone asked for an older version of the package. Oh, and UVersion's are a total order; there are no branches. But Universes is supposed to be a playground. Feel free to put together an alternate universe browser that follows whatever dependency rules you like!!
Hi Lex and all!
Just wanted to mention that I and Stephan have moved our exchange off list and that the status is as follows:
- We have agreed on some common ground in our models. - We are focused on trying both our models, either in parallell or merged somhow.
Just so you know.
Now, some remarks:
lex@cc.gatech.edu wrote: [SNIP]
One aspect of baking in dependencies into version numbers is that people may well end up not wanting to use the dependency-accurate version numbers as their normal version numbers. If am working on Chuck II Son of Chuck(y), then I just want version numbers that go 2.0, 2.1, 2.2, etc., but those will not capture the dependency information accurately. This makes me wary of capturing *accurate* dependency information in the versions, though it may still be that the *intent* of the dependencies can still be encoded in them.
Yes, our current "agreement" on this is to encode it in a separate field called "compatibility code". And it will not be entered as a number, Stephan has boiled it down to a series of yes/no questions.
We are also refining it so that *I* also am satisfied with it - this means both our approached will be able to use it.
Along these lines, note that a conflict can be detected after the version number has already been decided. What happens in that case? If the version number only captures intent then there is no problem. If it is supposed to be accurate, then that seems impossible.
Yes, but as I said - we put it in a separate field. You still have the manual version field and the automatic version number too.
regards, Göran
PS. Lex, I will look into your postings etc too. Just need time. I have also exchanged some emails with Julian offlist discussing the "multi server issues". I am still thinking. I have revised my plans for that a bit (and note that for me *that* stuff has been moved into the future *after* dependencies) but haven't decided anything yet.
Hello Lex and other interested readers,
first please excuse my late answer: but there is so much to do and the things are very much in flux. Currently I'm mostly thinking and writing: the intesting thing is, that I'm writing docu about features of the planned system *before* any coding! But this is very important in this area, since if some dependency resolving system should have a chance to be accepted, people have to understand, how to deal with it.
lex@cc.gatech.edu wrote:
Hey Stephan, I enjoyed reading this. By all means continue to explore in this direction and to refine your essay, so that people new to the topic can get going quickly!
Thanks for the motivation!
Here are a few quick thoughts on it.
First, if I'm not mistaken, your capabilities model seems to boil down, for each package, to this form:
(cap1 AND cap2 AND cap3) IMPLIES (cap4 AND cap5) (cap6 AND cap7) IMPLIES (cap8 AND cap9 AND cap10)
Is this correct? You'd ask the installer tool to find a way to get you cap10, and then it would trace backwards through all the packages to know about to see if it can satisfy this.
Yes. But the Caps can be hidden, so that the user asking for installing a package does not see them. And the mechanism allows to express more complex dependencies.
Do you have any concrete examples where these transformers would be useful, Stephan? It would help in thinking about them.
Good question ;-)
In general I want to have a mechanism which *allows* expressing complex dependencies as a backend, without beeing forced to go into nasty technical details in the frontend, at least for expressing simple dependencies.
A concrete example is the possibility to express ORed dependencies via logical Caps in extra Transformation rules: something like I need package SocketStuffFastButRisky in this OR package SocketStuffSlowButReliable in that version (assumed that both are OK for me). But this is just the tip of the iceberg...
One aspect of baking in dependencies into version numbers is that people may well end up not wanting to use the dependency-accurate version numbers as their normal version numbers. If am working on Chuck II Son of Chuck(y), then I just want version numbers that go 2.0, 2.1, 2.2, etc., but those will not capture the dependency information accurately. This makes me wary of capturing *accurate* dependency information in the versions, though it may still be that the *intent* of the dependencies can still be encoded in them.
As Göran has already written, compatibility code and version number are separated.
After some (to be honest many) thoughts and monster mail discussions with Göran, current state of the art (note: from my POV; I haven't gotten Görans feedback regarding it so far (it is *very* new)) is visible in http://minnow.cc.gatech.edu/squeak/3792 , a Swiki page about so called 'compatibility codes' (CCs). Note: this page is under construction, so many things already in my head are missing, but what is in there should reflect my current thinking accurately.
Though there is a 1-to-1 relation to some constant version number, in opposite to the latter the CCs may be *changed* afterwards. So there is a good reason to keep them separated (this insight is quite new to me, before I've thought they could be used as version number). The opportunity for these changes is necessary in a world not perfect...
Along these lines, note that a conflict can be detected after the version number has already been decided. What happens in that case? If the version number only captures intent then there is no problem. If it is supposed to be accurate, then that seems impossible.
There are some ways to go (writing more into this direction at the 'compatibility codes' Swiki page is in the queue):
1. a bug would be fixed resulting in a new package release without this conflict, having a new CC: then the package could be installed;
2. the package wouldn't be changed, but its CC would be corrected (not the *constant* version number): the dependency resolver would avoid it then;
3. a conflict would be expressed explicitely in the Transformation rules (e.g. if a package maintainer is on holidays).
The last variant is the worst, since it requires extra rules for describing the exception from the wished behavior.
Note: there may be more variants.
- The package name combined with the major number could
be seen as the - technically and socially - needed new name.
Now - after the separation of compatibility code and version number - I would write: "The package name combined with the minor <'minor' may change> compatibility code number could be seen as the - technically and socially - needed new name."
You beat me to it. For packaging purposes, it gives you a small simplification if Chuck1 and Chuck2 are simply different packages if they are in fact incompatible. Additionally, putting them in separate packages means that you can arrange for *both* of them to be installed simultaneously....
If they don't conflict! One important thing not to forget: that the API is compatible does *not* mean, that the implementation is compatible, too!
('WebBrowserStandard_2.2', "minor release" 'WebBrowserStandard_2.2.1', 'WebBrowserStandard_2.2.1.1', 'WebBrowserStandard_2.2.1.1.1')."
Keep in mind, everyone, that accurate dependencies require extra work to happen somewhere.
I know ;-)
I hope stuff like the above does not need to be maintained by humans!
I have good hope, that it is possible to hide it in most cases.
And anyway, it makes me wonder in a lot of cases if it is not simpler to *fix* a conflict between two packages and upload a new version of the package, as opposed to trying to accurately report that Chuck 1.28 is incompatible with RB 2.11.
Yes, of course: variant 1. above.
Note that we have to deal with bugs anyway, and that an incompatibility can often be considered simply a bug.
Correct.
But here again: we *need* a versioning policy, which people are following!
Yes, that sounds right to me. Bugfixes happen, and we should have a system that can deal rationally with them.
Versioning policies would let you easily (hopefully) express the claims about compatibility that people often have mentally.
I (and also Göran, of course) try to do my best to reach the 'easily'.
. Since we are faced with a
combinatorial explosion here, hints regarding stable/unstable/bleedingEdge/etc. from the package maintainers would be helpful though...
I suggest we solve this through the idea of package universes. You simply live in a bleeding edge universe or a stable one. I don't see how you can combine a bleeding edge package and a stable package and get something meaningful. The result is not stable, for sure.
There's no explosion in practice. The simple "Stable" versus "Unstable" goes a long way.
Here I have a question following in another thread.
There also may be a logical package WebBrowser which just depends on one of them. But who makes the choice? This is similar to the question, who decides which stuff is worth enough to come into the official distribution.
Exactly!! There is no one answer to this. A package universe can be viewed as *one* answer to the question "what packages exist?"
It would be nice to have a detection of class extension conflicts.
This sounds like a job for the package loader and unloader, and if I'm not mistaken Monticello will do this already.
But here it is important, that the dependency resolver sees this conflicht *before* installing! That is the problem.
Don't try to be too perfect: KISS! I have to admit that KISS seams to be quite complex in this domain...
I am not convinced. If we restrict attention to individual package universes, then simple unversioned dependencies seem to be sufficient. It is worth trying to improve on them, but I don't see why we *have* to improve on them. How much talking *about* our packages do we really want to be doing, as opposed to actually improving the packages themselves? :)
It depends on what you want: if you want to have packages accessible in different versions (for simplicity let's assume different stable versions) and packages requiring different of these stable versions (e.g. they need different APIs) accessible in one catalogue, then the world is not so simple.
-Lex
PS -- There is a non-trivial UVersion in the Universes prototype I posted, but it is way shy of the kind of stuff you are talking about. I have made -- KISS remember :) -- the assumption that incompatible packages have different names, and that newer versions of packages are prefered even if someone asked for an older version of the package. Oh, and UVersion's are a total order; there are no branches. But Universes is supposed to be a playground. Feel free to put together an alternate universe browser that follows whatever dependency rules you like!!
Thanks for the pointer: but I don't have time to follow it... But I *have* read the corresponding post!
Greetings Stephan
First, if I'm not mistaken, your capabilities model seems to boil down, for each package, to this form:
(cap1 AND cap2 AND cap3) IMPLIES (cap4 AND cap5) (cap6 AND cap7) IMPLIES (cap8 AND cap9 AND cap10)
[...]
Yes. But the Caps can be hidden, so that the user asking for installing a package does not see them. And the mechanism allows to express more complex dependencies.
Well that was my question. Can you give an example of something more complex than what I showed?
Do you have any concrete examples where these transformers would be useful, Stephan? It would help in thinking about them.
Good question ;-)
In general I want to have a mechanism which *allows* expressing complex dependencies as a backend, without beeing forced to go into nasty technical details in the frontend, at least for expressing simple dependencies.
You may want to consider this strategy carefully. If the point is to build a useful system, then you *don't* want it to be able to do absolutely everything. If your scheme does not support something, but it's something no user desires, then the users still do not have to do complicated things in the front-end.
A concrete example is the possibility to express ORed dependencies via logical Caps in extra Transformation rules: something like I need package SocketStuffFastButRisky in this OR package SocketStuffSlowButReliable in that version (assumed that both are OK for me). But this is just the tip of the iceberg...
These examples do not require transformers, and do fit the slimmed-down model I described above. Both of the two socket packages can each provide "SocketStuff".
Have you run across an example where a package provides extra abilities whenever some other package is around? I've seen this in code a lot -- for example, you can storeString an array iff its elements understand storeString -- but I am having trouble thinking of an example involving packages.
One aspect of baking in dependencies into version numbers is that people may well end up not wanting to use the dependency-accurate version numbers as their normal version numbers. If am working on Chuck II Son of Chuck(y), then I just want version numbers that go 2.0, 2.1, 2.2, etc., but those will not capture the dependency information accurately. This makes me wary of capturing *accurate* dependency information in the versions, though it may still be that the *intent* of the dependencies can still be encoded in them.
As Gsran has already written, compatibility code and version number are separated.
Cool. The checkboxes he describes sound very easy to use.
But it does not address the main question I was getting at, that is about *intended* compatibility versus measured compatibility. If you are willing to drop back to encoding *intended* compatibility, then you can greatly simplify your work. For example, with intended compatibility, you would never need to change the compatibility information retroactively.
Along these lines, note that a conflict can be detected after the version number has already been decided. What happens in that case? If the version number only captures intent then there is no problem. If it is supposed to be accurate, then that seems impossible.
There are some ways to go (writing more into this direction at the 'compatibility codes' Swiki page is in the queue):
- a bug would be fixed resulting in a new package release without this
conflict, having a new CC: then the package could be installed;
- the package wouldn't be changed, but its CC would be corrected (not
the *constant* version number): the dependency resolver would avoid it then;
- a conflict would be expressed explicitely in the Transformation rules
(e.g. if a package maintainer is on holidays).
The last variant is the worst, since it requires extra rules for describing the exception from the wished behavior.
Note that in case 1, the dependency information is less specific than the known compatibility information. The recorded dependency information will not only let people install the new package, but they will also allow them to install the old incompatible one.
In cases 2 and 3, you are fixing the dependency information, but it means that people who have the package already installed are going to enter a state where their dependencies are not met.
You beat me to it. For packaging purposes, it gives you a small simplification if Chuck1 and Chuck2 are simply different packages if they are in fact incompatible. Additionally, putting them in separate packages means that you can arrange for *both* of them to be installed simultaneously....
If they don't conflict! One important thing not to forget: that the API is compatible does *not* mean, that the implementation is compatible, too!
Yes. It happens sometimes, though. An example from Debian would be the packages "squeak-image3.4" and "squeak-image3.5". I have carefully arranged that these can be installed simultaneously. All my care would have been for naught, however, if I had called them the slightly different "squeak-image-3.4" and "squeak-image-3.5", because then the installer tool would consider -3.5 to be an upgrade of -3.4.
In short, it seems helpful to make your theoretical name of "name plus first major verion", in fact be the *real* name of the package.
And anyway, it makes me wonder in a lot of cases if it is not simpler to *fix* a conflict between two packages and upload a new version of the package, as opposed to trying to accurately report that Chuck 1.28 is incompatible with RB 2.11.
Yes, of course: variant 1. above.
Be careful not to speak too fast -- I inserted an "easier" in there. If you really think that variant 1 is all you need, then you will have less work to do because you can just assume people will tend to grab newer packages in preference to older ones. Variant 1 means you can fix dependency problems just like you fix any other bugs: you simply release a new version.
It depends on what you want: if you want to have packages accessible in different versions (for simplicity let's assume different stable versions) and packages requiring different of these stable versions (e.g. they need different APIs) accessible in one catalogue, then the world is not so simple.
By definition, an auto-installer will *not* want to have everything accessible in one catalog. An auto-installer should only show those packages which have a good chance of being installable, which will typically be a subset.
-Lex
lex@cc.gatech.edu wrote:
First, if I'm not mistaken, your capabilities model seems to boil down, for each package, to this form:
(cap1 AND cap2 AND cap3) IMPLIES (cap4 AND cap5) (cap6 AND cap7) IMPLIES (cap8 AND cap9 AND cap10)
[...]
Yes. But the Caps can be hidden, so that the user asking for installing a package does not see them. And the mechanism allows to express more complex dependencies.
Well that was my question. Can you give an example of something more complex than what I showed?
More to this point below.
Do you have any concrete examples where these transformers would be useful, Stephan? It would help in thinking about them.
Good question ;-)
In general I want to have a mechanism which *allows* expressing complex dependencies as a backend, without beeing forced to go into nasty technical details in the frontend, at least for expressing simple dependencies.
You may want to consider this strategy carefully. If the point is to build a useful system, then you *don't* want it to be able to do absolutely everything.
The core mechanism should be as flexible as possible: therefore you have to model at a level abstract enough. This also should lead to 'elegant' concepts: - no redundancy, - as less elements as possible, - well defined, - good terms (language is important!).
But it should be possible to do complex things with them...
If your scheme does not support something, but it's something no user desires, then the users still do not have to do complicated things in the front-end.
1. There is a difference between - more or less complex backend mechanisms, needed for providing some functionality, and - the usage of this functionality in the frontend, beeing as simple as possible.
2. I'm thinking of making a dependency mechanism not only usable by a package management system.
A concrete example is the possibility to express ORed dependencies via logical Caps in extra Transformation rules: something like I need package SocketStuffFastButRisky in this OR package SocketStuffSlowButReliable in that version (assumed that both are OK for me). But this is just the tip of the iceberg...
These examples do not require transformers, and do fit the slimmed-down model I described above. Both of the two socket packages can each provide "SocketStuff".
Agreed. But if this hasn't been expressed in the package version provides, it is possible to introduce a logical package providing "SocketStuff" *afterwards* without changing the provides of the original package versions.
Have you run across an example where a package provides extra abilities whenever some other package is around? I've seen this in code a lot -- for example, you can storeString an array iff its elements understand storeString -- but I am having trouble thinking of an example involving packages.
Also see above. A simple example: 'MozillaFull' as logical 'package' dependent from the real packages 'Mozilla_WebBrowser' and 'Mozilla_MailClient'. Could also be modeled as conditioned provide of one of the real packages, if the other one exists (but this seems to be more complicated).
One aspect of baking in dependencies into version numbers is that people may well end up not wanting to use the dependency-accurate version numbers as their normal version numbers. If am working on Chuck II Son of Chuck(y), then I just want version numbers that go 2.0, 2.1, 2.2, etc., but those will not capture the dependency information accurately. This makes me wary of capturing *accurate* dependency information in the versions, though it may still be that the *intent* of the dependencies can still be encoded in them.
As Gsran has already written, compatibility code and version number are separated.
Cool. The checkboxes he describes sound very easy to use.
But it does not address the main question I was getting at, that is about *intended* compatibility versus measured compatibility. If you are willing to drop back to encoding *intended* compatibility, then you can greatly simplify your work. For example, with intended compatibility, you would never need to change the compatibility information retroactively.
It is always 'intended' compatibility what a package maintainer states; it is a human being! After thinking a while about the CC (compatibility code) issues, I've come to the point, that there a good cases, where it makes sense to change them *afterwards*, and to recompute the CCs for newer versions (which are depending on the change). See 'compatibility code' Swiki page http://minnow.cc.gatech.edu/squeak/3792 for incrementally computing CCs.
Along these lines, note that a conflict can be detected after the version number has already been decided. What happens in that case? If the version number only captures intent then there is no problem. If it is supposed to be accurate, then that seems impossible.
There are some ways to go (writing more into this direction at the 'compatibility codes' Swiki page is in the queue):
- a bug would be fixed resulting in a new package release without
this conflict, having a new CC: then the package could be installed;
- the package wouldn't be changed, but its CC would be corrected
(not the *constant* version number): the dependency resolver would avoid it then;
- a conflict would be expressed explicitely in the Transformation
rules (e.g. if a package maintainer is on holidays).
The last variant is the worst, since it requires extra rules for describing the exception from the wished behavior.
Note that in case 1, the dependency information is less specific than the known compatibility information. The recorded dependency information will not only let people install the new package, but they will also allow them to install the old incompatible one.
It depends on how the dependencies will be computed: this may be dependent on some policy. If you give the CC an important role in this play the behavior described above is possible.
In cases 2 and 3, you are fixing the dependency information,
but it means that people who have the package already installed are going to enter a state where their dependencies are not met.
This is alway true, if there is a conflict! Which has been the assumption for this scenario.
You beat me to it. For packaging purposes, it gives you a small simplification if Chuck1 and Chuck2 are simply different packages if they are in fact incompatible. Additionally, putting them in separate packages means that you can arrange for *both* of them to be installed simultaneously....
If they don't conflict! One important thing not to forget: that the API is compatible does *not* mean, that the implementation is compatible, too!
Yes. It happens sometimes, though. An example from Debian would be the packages "squeak-image3.4" and "squeak-image3.5". I have carefully arranged that these can be installed simultaneously. All my care would have been for naught, however, if I had called them the slightly different "squeak-image-3.4" and "squeak-image-3.5", because then the installer tool would consider -3.5 to be an upgrade of -3.4.
In short, it seems helpful to make your theoretical name of "name plus first major verion", in fact be the *real* name of the package.
The model in my mind has: - package name, - one CC for each version for each compatibility measurement universe, - constant version numbers.
This shouldn't be put into one name string IMHO.
And anyway, it makes me wonder in a lot of cases if it is not simpler to *fix* a conflict between two packages and upload a new version of the package, as opposed to trying to accurately report that Chuck 1.28 is incompatible with RB 2.11.
Yes, of course: variant 1. above.
I've read '*fix*' as bug fix here, and made my argument accordingly.
Be careful not to speak too fast -- I inserted an "easier" in there. If you really think that variant 1 is all you need,
I don't think so, as I've written above (describing more variants). Variant 1 is just the best for a *bug* fix.
then you will have less work to do because you can just assume people will tend to grab newer packages in preference to older ones.dd
Variant 1 means you can fix dependency problems just like you fix any other bugs: you simply release a new version.
This is possible for fixes of *bugs*. If there is e.g. an API incompatibility: what to fix there?
It depends on what you want: if you want to have packages accessible in different versions (for simplicity let's assume different stable versions) and packages requiring different of these stable versions (e.g. they need different APIs) accessible in one catalogue, then the world is not so simple.
By definition, an auto-installer will *not* want to have everything accessible in one catalog.
Disagreed, as it is written. Agreed, if you have filters working before and/or with the dependency resolver and you mean the state *after* filtering. Before filtering *everything* may be accessible in *one* catalogue.
An auto-installer should only show those packages which have a good chance of being installable, which will typically be a subset.
Agreed.
Greetings Stephan
-Lex
Lex and others,
a few additional thoughts (after thinking about it a while ;-) ):
I wrote:
...
lex@cc.gatech.edu wrote:
...
First, if I'm not mistaken, your capabilities model seems to boil down, for each package, to this form:
(cap1 AND cap2 AND cap3) IMPLIES (cap4 AND cap5) (cap6 AND cap7) IMPLIES (cap8 AND cap9 AND cap10)
Is this correct? You'd ask the installer tool to find a way to get you cap10, and then it would trace backwards through all the packages to know about to see if it can satisfy this.
Yes. But the Caps can be hidden, so that the user asking for installing a package does not see them. And the mechanism allows to express more complex dependencies.
I have left out one important point: time is missing here!
What does this mean? Let me give an example.
Assumed you want to install a patch to another package: obviously it needs the existence of the other package to be applied (otherwise it wouldn't be a patch). So there is a *pre*requirement of having the package installed *before* the patch.
Another scenario: say there is a library needed for some package, which doesn't have to be there for installing purposes of the package (since they are independent enough): then its OK first to install the package, and after that the library, to get a working system *after* the install process. There is a *post*requirement which has to be fulfilled *after* the installation as a whole, but not before installing the package.
This is an example for the difference between incremental and non-incremental installation.
The non-incremental variant is more comfortable for the dependency resolver: it allows to temporarily remove a requirement as long as it is provided again at the end of the installation process. In the incremental case such a behavior would fail in many cases.
In the Boolean logic example above time is not an issue: but the world is more complex than this example suggests.
Greetings Stephan
...
Lex,
lex@cc.gatech.edu wrote:
...
. Since we are faced with a
combinatorial explosion here, hints regarding stable/unstable/bleedingEdge/etc. from the package maintainers would be helpful though...
I suggest we solve this through the idea of package universes. You simply live in a bleeding edge universe or a stable one. I don't see how you can combine a bleeding edge package and a stable package and get something meaningful. The result is not stable, for sure.
There's no explosion in practice. The simple "Stable" versus "Unstable" goes a long way.
The paragraph in my original post has been:
* But testing is nevertheless important: it would be very good to have * many automatically runnable tests to classify package combinations as * stable/unstable/bleedingEdge/etc.. Since we are faced with a * combinatorial explosion here, hints regarding * stable/unstable/bleedingEdge/etc. from the package maintainers would be * helpful though...
So the 'combinatorial explosion' means *testing* of package combinations here: and there is such a thing *even* in universes separating stable/unstable/etc. with only one kind of packages. But putting all in one universe *without* labeling them (as suggested) makes the problem much more worse. That has been my point here.
Greetings Stephan
...
squeak-dev@lists.squeakfoundation.org
-
goran.krampe＠bluefish.se -
lex＠cc.gatech.edu -
stan＠stanheckman.com -
Stephan Rudlof