Hi Göran and all!
Göran, thanks for your constructive reply.
After reading it once, I have the impression, that this could lead to an interesting debate *improving* the concepts.
<Some time later> I'm a little bit tired now... ;-)
I've sitted one night to write this answer and many things have become clearer while thinking about and writing the answers followed by iterated thinking about previously treated parts. I've been faced with misunderstandings and hopefully my thoughts about your thoughts are coming close to reality now... So I suggest to read this to the end before hitting the reply button and starting writing, many things should become clearer while going this long way and if gotten the whole picture...
Göran, please don't feel forced to reply fast, I really understand if this takes a while...
But now let's go!
At first I want to clearify some definitions to avoid confusion, which has arosen on both sides, I think.
I've used the term Package (uppercase 'P') *always* in the narrower sense of 'one package-version combo', e.g. Package WebBrowser_2.1.3.4.1. There is admittedly at least one exception (the headerline 'Package WebBrowser' below obviously denotes a bunch of versions, but it is just 'packages' written as section headerline with uppercase first letter).
And I've used the term 'package' context dependent: sometimes as Package, but also sometimes more general, e.g. package WebBrowser. I have found this more general use in *opposite* to your definition, which seem to match the definition of DEPS Packages above, as I see it now (at least these interpretations have led to a consistent picture in my head).
I think we have an overloading of the term 'package', which leads to confusion.
Is 'package' for you a shortcut for a package-version combo, which would be the same as a DEPS Package above? Do you mean with 'release' one of many packages (package-version combo), stating that something important has changed in the code, of some kind of "Don't know how to avoid the word 'pack...'" doing something (e.g. a WebBrowser "Don't know...")?
Another problem: I've also been confused by the term 'release': I would expect to have only releases - tagged stable versions in a version control sense - at SM, but you seem to have another definition/policy/whatever there...
I hope this helps for reading the rest (there is at least one highlight waiting for you :-) ):
goran.krampe@bluefish.se wrote:
Hi Stephan and all!
I got a bit tired of writing long postings, but since I definitely don't want to loose the opportunity to cooperate with Stephan here I will take the time to reply to this one know.
Before I start - Stephan - don't be offended if I don't agree with you now. :)
No problem.
And I will try to snip parts that I have no comment on.
Good (I have asked for it).
Stephan Rudlof sr@evolgo.de wrote: [SNIP]
Technical Terms ###############
Caps
There are Capabilities (Caps) each describing the capability to do something. Caps mostly depend on other Caps, which are prerequisites of them. In the following Caps means Capabilities or a union of them.
This "Caps" layer is an extra abstraction layer that I am not sure about. I understand the theory behind it - but it feels a bit on the overly complex side, at least if it is the primary mechanism. If it was an additional abstraction that one could select to use or not, then it would be another matter.
But I will read on first... :)
Transformations
Transformations describe how it is possible to change a set of Caps: they may - virtually at first - add or remove Caps or both. They transform one set of Caps into another one. To do their work they perform Actions.
Inputs are I.1. requires: required Caps, and I.2. conflicts: conflicting Caps, outputs are O.1. provides: newly generated Caps, and O.2. deletes: deleted Caps. Properties:
- provides are const sets, but
- deletes are configuration dependent.
Outputs are expressed as
- set of (named) Caps.
Inputs are expressed as
- set of (named) Caps (as for outputs), or
- as a Block
- taking a Caps set as an argument, returning
- true, if the requirement/conflict has been fullfilled,
- false, otherwise.
Hmmm, ok - at first look this seems again pretty... advanced. Or complex.
To Blocks as input
It is computational expensive to use Blocks as input, since in principle all existing Caps and those somehow generatable by whatever Transformation combination (!) have to be checked.
An - important, I think - improvement for the case of expressing conflicts may be to divide the Blocks into two classes (later there may be more):
- Blocks just checking conflicts with Caps generated additionally by
the packages generating the requirement Caps, under the precondition, that the requirement Caps are given as set. 2. Other Blocks.
See section "To Lex's example" below for a motivation of Blocks as arguments here and the suggested improvement.
Ok, this part isn't at all clear to me at this point. But it is only a reflection so...
Boolean logic
Require sets model a logical AND of requirements. Logical ORs - and therefrom Boolean logic - can be modeled via additional Transformations with newly introduced Caps names. This is far more efficient as trying to do this in one Transformation using the Blocks mechanism!
Ouch. Ok, my first reaction is "NO!". :) I know it is tempting to add the capability to use boolean logic to express dependencies/scenarios or whatever - and Debian uses it - but personally I have elected to NOT go there because the complexity seems to me to be unwarranted.
If I've understood you correctly below, you also have AND logic (multiple requires) in your logic. A logical Caps just extends this to also have an OR.
How would you model something like WebBrowserUsingApp requires: (WebBrowserStandard | WebBrowserNicerButSlow) ; without some kind of Boolean logic?
With Caps there are two ways to go in this example.
The first one assumes there is no logical Caps 'WebBrowser'. Then there is a need to introduce two new - very simple - Transformations
requires: ('WebBrowserStandard') | V provides: ('WebBrowser').
requires: ('WebBrowserNicerButSlow') | V provides: ('WebBrowser').
and to require 'WebBrowser' in the 'WebBrowserUsingApp' generating Transformation.
The second alternative is chosen below: there the idea of having a logical 'WebBrowser' has been at 'package providing/deleting Transformation' creation time of WebBrowserStandard and WebBrowserNicerButSlow, so this Caps 'WebBrowser' has just been added to their provides then. Note: Puh, what a huge word combo is "at 'package providing/deleting Transformation' creation time" ! But I've wanted to be as precise as possible. I think we need some definitions, but not now...
The good think here is, that it is possible to *add* such a logical choice *afterwards*, if it hasn't been done at writing the Transformations for one of the Browser packages, without the need to change them! So later there can be better and better Transformation rules, with different originators: there needn't be one central instance (without such a mechanism you had to change the orginal Transformations in direction of the second alternative.
Note: This example doesn't forbid to ask the user for a decision if the dependeny resolver comes to this point. But this is an UI issue.
To the complexity issue: What is complex here?
I don't see any problems into traversing a graph of nested AND/OR connected requires and trying the alternatives if there is an OR (the AND exists in every non-trivial dependency logic, I assume).
Am I missing something?
But ok, shutting up and reading on... :)
Outlook to packages
A package removal can be modeled by transforming the all over package Caps as requires into deletes of this and all other package related Caps.
Don't understand, but not important at this point I guess.
How to model dynamic provides?
The restriction 'provides are const sets' eases some things, but it is not a limitation AFAICS.
Example: Installing a package WebBrowser may result into more Caps if some other package SuperFonts is there. This can easily be modeled by a *logical* (no Action) package Transformation describing the all over functionality of the packages combo, with
- requires: Caps ('WebBrowser', 'SuperFonts'), and
- provides: something like ('WebBrowserWithSuperFonts').
Ok, at this point I am thinking - perhaps this Caps layer should be something on top of SM? This may mean that your model could be a layer on top of mine. Just a feeling at this point.
Note
It's questionable, if naming both set and Block inputs the same is a good idea, but I want to stop for now...
Actions
Actions are commands leading to a change of the (set of) Caps in the system. If triggered, they add and/or remove one or more Caps in reality.
Algorithms ############
Before there is no good model of the domain, it is not urgent to think about algorithms. I don't fear not to find some...
A few things I'm currently expecting to be faced with: Easy:
- depth first traversing of dependency graphs,
- back tracking with detection of cycles.
Not so easy:
- combinatorial explosion, if Blocks as inputs for Transformation
requires and conflicts come into play (but section 'Transformations->To Blocks as input' contains a first idea for embanking it).
Yes, the reason for me starting to write code was that I wanted to explore at least some of the algorithms involved.
Application to packages #######################
Packages
A Package provides Caps: these are representing some things to be made with or by the Package (interfaces, commands). There may be more than one Caps per Package!
Since this would be a thing changing over time I would presume that actually a package *release* provides Caps?
After my definitions: - *Each* DEPS Package provides a *set* of Caps. - Some sets of package-version combos (or Packages) are sharing the same Caps (one or more). - A set of package-version combos belonging to the same e.g. WebBrowser general package shares one or more of the provided Caps! At least after the proposed versioning number scheme below...
So a 'package *release*' - as I've understood you - has the following properties in DEPS notation: - it provides a set of Caps, - it should have a change in one of the higher ranked version numbers (to express the semantically important code change), - it provides a Caps (in the set of its provided Caps) shared by a set of Packages (all are providing it) containing this release and all sub-releases (this could be used to express a dependency or conflict to a set of Packages at release compatibility level).
I think this will become clearer if you read further to more versioning number examples and - neat outlook - how to map them to categories and vice-versa!
A Package needs Caps provided from other packages (requires).
Same thing here. In my model this extra "redirection layer" is not present - there a release has tested configurations and each configuration more or less lists the other package *releases* that are needed for the release to work.
I see a problem here with your approach: you are faced with a combinatorial explosion. One package release requiring say 4 other packages each existing in say 4 releases leads to 4 ^ 4 = 256 configurations...
Let's assume, there are 10 tested configurations. Then you have to switch to one of the 246 untested ones if there is only one other general package requiring just one of the 4 requires in another release (assumed that different versions cannot coexist).
To help me understand: Why aren't you seeing a problem here?
If I've understood you correctly, you would try to come through this via the compatibility level categories combined with some rules for controlled deviation from the tested configurations, but currently I have serious stomaches here.
Which made me think perhaps this capability layer can be "on top" of my model.
More to this point below.
A Package providing some Caps may conflict with another Package (providing other and/or the same Caps).
To be able to express these conflicts at Caps level it is necessary that all packages are mutually different in at least one Caps! But this is easy to accomplish, if there is just one Caps named so, that it contains the - unique - name of them.
Note that a Cap could easily be modelled using SMCategories. That is the point with SMCategory - it is a unique canonicalized "tag" that can be attached to SMObjects.
I am not sure what you are saying though in the last part. Are you saying that two different packages (releases) can not provide the exact same Caps?
Exactly. And they don't after the proposed versioning scheme.
Some explanation, though: here Caps in the sense of union of Caps is meant. E.g. general package Foo in version 4.4.3.2.3 provides standard Caps (via a corresponding Transformation) ('Foo', 'Foo_4', 'Foo_4.4', 'Foo_4.4.3', 'Foo_4.4.3.2', 'Foo_4.4.3.2.3') ; another minimal changed version 4.4.3.2.4 of the same general package provides standard Caps ('Foo', 'Foo_4', 'Foo_4.4', 'Foo_4.4.3', 'Foo_4.4.3.2', 'Foo_4.4.3.2.4') . In this example both versions share all, but the last Caps.
Note: this representation is made from/for a semantic point of view; in an implementation one #Foo_4.4.3.2.4 may be sufficient to represent all standard Caps (but there may be more).
Why not?
To be able to differentiate between two different package-version combos at the Caps level. This is needed to be able to express a conflict with just this special seldom used one version of package Foo (e.g. expressed as conflict with Caps 'Foo_4.4.3.2.3').
A conflicting Package has to be removed (this can be difficult) before installing a new one, leading into a reduction of its provided Caps (there may be logical ones left, which are also provided by another package). There may also be Packages, which only remove Caps without adding new ones (system shrink)!
Versions
This section has become very long and is very crucial regarding its social aspects.
There *have* to be policies to be followed by the package maintainers. I'm currently thinking of version numbers like 1.1.3.4.2 #(1 1 3 4 2) 2.7.3.1.1 #(2 7 3 1 1) , where a change in one of the numbers means:
- incompatible, changed interface semantics, called major package
number: the package is expected to be incompatible with older or newer ones having another number here (at this position);
- downwards compatible, called minor package number: the package has
changed, and their may be new features, but it is expected to work with software written for lower numbers here (but with same 1. number, of course);
- downwards *and* upwards compatible: the package has changed, but
there are *no* new features here (but it may be faster for example), so it is expected to work with software written for higher numbers here (but with same 1., 2. numbers, of course);
- compatible: 'small' fixes without changed functionality in the sense
above (also see 'Classification issues' below);
- no code changes at all: no changed semantics, but e.g. improved class
comments.
After this scheme, a version number a.b.c.d. for a package P corresponds to the Caps 'P', 'P_a', 'P_a.b', 'P_a.b.c', 'P_a.b.c.d', 'P_a.b.c.d.e'. Caps 'P_a.b' - normally - includes *every* Caps 'P_a.x', x < b, since the functionality/interface has just been smaller then. The longer numbers are for expressing conflicts and requirements in a world not being perfect. This means, we finally have defined a relationship between Caps and version numbers! And it should be clear now, why there is this funny plural with the 'Caps' term: it is a normal case for a package-version combo, that it provides a *set* of Capabilities!
I am not comfortable with adding yet another version numbering scheme. It seems simply too complex to me.
I have more feared, that it could be too simple ;-) As I've written in another Mail to Stan: "The granularity of the proposed scheme is fine, but better too fine as too simple: nobody is forced to publish versions only changing in lower ranked version numbers, but it is possible. And it is also possible to be very fine granular in expressing conflicts with other package versions: from a singular version to larger and larger sets of versions."
I imagine one simple dialog asking a package release publisher five or less questions for computing a new version number (for all increase rules applies, don't change higher ranked version numbers, reset lower ranked ones (similar to numbers ;-) )):
1. Do you expect, that most code running with the previous version, does *not* work anymore with this new version? yes -> increase 1., finished; no -> continue...
2. Do you expect, that most code running with the previous version, continues to work with this new version? yes -> increase 2., finished; no -> continue...
3. Do you expect, that most code to be written for running with this new version runs with the previous version, too? yes -> increase 3., finished; no -> continue...
4. Do you expect, that 3. applies *and* have you made a small fix without adding new functionality? yes -> increase 4., finished; no -> continue...
5. Do you have *not* changed any code at all (just comments, etc.), so that's *impossible* that other code breaks? yes -> increase 5., finished.
Of course with 'version' is meant a published/public version *** thought to be used by others *** here, so these questions don't apply to 'private' versions (probably they shouldn't go into the dependency system at all, but manually installed).
It should be possible to answer these maximal 5 questions for a package maintainer (beeing the target group here), otherwise I wouldn't like to have his/her package... ;-)
SMPackageReleases has today one manually edited field "version" and one automatic version number which can't be edited. I want to express the "level of change" in one release compared to the previous release using categories instead. And I have created a first shot at these categories in SM and made it mandatory to select one of those when you register a new release.
If you'd choose the categories along the questions above, the DEPS version numbers could be computed automatically (coming again as highlight below)...
Some prospects for convergation here :-)
Classification issues
It seems to be difficult to classify fixes (4.) correctly. Fixes definitely change the formal semantics, so one has a good reason to rank them higher and exchange them with 3. for example. A version number change in 3. normally doesn't change the formal semantics (but may introduce new bugs as a change in 4. as well, changing the formal semantics then).
But if a 'small' fix just makes the package working with some seldom used other package being the only one running into the bug, then the classification as 4. makes sense (3. may introduce new bugs affecting many packages). But if the fix removes some serious problems affecting many other packages, it may as well classified as 2., since there is good reason to view this as an important upgrade.
A similar argumentation could be made for changes in the functionality classified as 3.: a piece of code just running much faster without changing the formal semantics, could open up new use cases. This may as well earn a classification in 2..
Currently I think the intelligence of the package maintainers is questioned here, since 'it depends'...
I interpret this to mean that the classification is not a black and white thing
Yes.
- but that it helps a lot to have.
I agree.
I have to think here. Without this classification there would just be standard Caps like e.g. for general package Foo in version 4.4.3.2.3: ('Foo', 'Foo_4_4_3_2_3') ('_' instead of '.' for not thinking they are made from real DEPS versions), and you wouldn't have any semantically meaningful relationship of the latter (version) Caps 'Foo_4_4_3_2_3' to Caps 'Foo_4_4_3_2_4' of the next version. So 'Foo_4_4_3_2_4' is equally 'far' away from 'Foo_4_4_3_2_3' than e.g. 'Foo_9_9_9_9_9'.
Automatically upgrading wouldn't work; so classification is *needed* here. The classification may be wrong, but this can be repaired via more/bigger Transformations and/or prefiltering (e.g. by disabling packages with a compatibility classification too optimistic (too pessimistic doesn't break anything)) of packages to be considered for Transformations at all.
Note to fixes
An important fix (changed version number 4.) can make the package work the first time with another one. It may make sense - especially if the major number recently has changed - to apply such a fix to an older version, too! This would lead to a new older package with a new number, of course.
Automatically upgrading
In general automatically upgrading should
- apply to all packages increasing their minor or lower ranked version
number,
- *not* apply to packages increasing their major version number.
A harvesting process could be introduced using this technical solution as follows: addditional classifications of packages as stable/unstable/etc. could prefilter the number of packages to be searched for fulfilling dependencies (this classification could also be realised by just putting them into classified - different - package directories).
I think exactly how auto upgrading will work will be heavily parameterized
Agreed.
and also ask the user for input.
Agreed, but with limitation: An end user just using Squeak only wants to have a successful installation with as less questions as possible (and the end user often wouldn't know how to answer).
And different people can have different engines
Yes. But there will be some default.
- so this isn't crucial
stuff to "agree on".
Please don't be so fast here: - as better the classifications, as better the automatically upgrading; - as worse the classifications, as worse the automatically upgrading!
This won't be changed by another parametrization or engine. Or did I miss a point here?
Why major version number 1. at all?
After this versioning policy automatically upgrading would (mostly) work from version to version with a *fixed* first version number. If the major number changes, there are incompatible versions (since we have defined it so!). This could lead to the question, why not to ommit this major number and to force the package maintainer to publish a new package with a different name instead.
Arguments for a major version number increase instead of a name change:
- There are cases for expressing the existence of some functionality at
a very high level independent from the major number: e.g. some package wants to have a package WebBrowserStandard for enforcing to give the user a chance of viewing its generated HTML files (without assuming any API for calling it itself): a newly created name (how to guess it for other package maintainers?) instead of a major number increase would make expressing this dependency impossible.
- Though an interface has changed, the functionality may be very
similar, and the package name should express functionality.
- The package name combined with the major number could be seen as the -
technically and socially - needed new name.
Personally I think the identity of a package should foremost be the identity of the code base evolving.
Sure, people can always elect to "fork" a package, by creating a new one
- but hopefully in the coming SM that supports branches in the release
tree such forks will be even more rare.
Increasing 1. can be considered as starting a fork (after the proposed DEPS versioning scheme).
While thinking about this there has come the following question into my mind: what is the difference between a versioning scheme needed for a package dependency system with *automatically* upgrading, and a versioning scheme needed for a version control system?
The main difference - as I currently see it - is, that - for the package dependency system, versioning should reflect the compatibility *categorization/classification*, which is a pure semantical thing; and - for the version control system, versioning just says, how far two versions are *technically* away (e.g. by applying patches), which as itself says *nothing* about the semantical changes.
Consequences
This is a client POV versioning policy. There may be others, of course: IMHO it's not so important which policy to have, but
*** It's very important to have a policy at all! ***
Otherwise a technically working dependency mechanism doesn't work for packages because of social reasons...
Technically this means to define some Caps <-> version number relation with an order of Caps corresponding to an order of version numbers like described above (or to use a similar mechanism).
*** Otherwise the technical solution *cannot* work! ***
But I assume we can still represent the "level of change" using categories instead of version numbers?
Here comes the promised *highlight* a little bit more detailed:
If each package-version combo gets categories corresponding to answers to the questions to package maintainers as written above: then we even have a *mapping* from categories to version numbers (incrementally computed) and vice versa (direct mapping).
Isn't that nice?
Installation of packages
In general there are
- pre-install,
- do-install,
- post-install,
- pre-remove,
- do-remove,
- post-remove,
Scripts (they may be empty). They are triggered by performing an (package install/removal) Action.
Normally the install phase results into adding all provided Caps and the remove phase into the removal of them. Potential conflicts have to be solved *before* installation!
Personally I think we should strive to get away from scripts - they just break. :)
Seriously.I would rather try to use a declarative approach as long as possible and then only add hooks for scripts if we find out it is really needed.
I didn't want to give a definition for 'script' here; this part has mainly been written down to show, where I see some reasonable hooks, whatever 'scripts' should be. Better had been to replace "Scripts (they may be empty)." by "Scripts (they often are empty)."
Scripts can also seriously trip us up and produce weird results that we will have a hard time tracking down.
Yes, of course.
Different kind of installs
There are different kind of installs:
- upgrading,
- full installation with a previous removal of an older version (if it
exists).
Yes, currently SM actually differentiates these but only MC has the capability of doing things "differently" when upgrading - and it figures that out on its own today.
Examples (for looking, if the concepts are working) ##################################################
Package WebBrowserStandard
'Package': normally I've meant the package-version combo, here the more general 'package' has just been written as headerline...
- Provides Caps 'WebBrowser' and more;
- needs Caps 'SocketStuff' and 'RenderingMachine';
- exists in versions 1.1.3.1.7, 2.1.2.3.1, 2.2.1.1.1;
- conflicts with package 'WebBrowserNicerButSlow' and some special
SocketStuff versions.
I note that you have the Caps on the package and not on the releases. This would mean they can't change over time, or rather - the releases can't - at a specific point in time - have different Caps.
They can! Via changed and/or newly added Transformations. Caps besides the standard Caps (since they stay forever) could be changed arbitrarily.
I also note you are expressing conflicts on package level, which also suffers from the same problem.
For me one package-version combo (hopefully the same as exactly one installable 'package') is the finest granular entity. It has a *set* of standard Caps; e.g. (not in ST notation)
('Foo', 'Foo_4', 'Foo_4.4', 'Foo_4.4.3', 'Foo_4.4.3.2', 'Foo_4.4.3.2.3')
You can express conflicts to just this one package Foo_4.4.3.2.3 via a conflict with (one of the standard) Caps 'Foo_4.4.3.2.3' only provided by this package. But it is also possible to express a conflict to some set of packages, e.g. by a stating a conflict with Caps 'Foo_4.4' many versions are meant, exactly those all providing this Caps (e.g. packages Foo_4.4.7.3.3, Foo_4.4.7.3.3).
Each package version combo WebBrowserStandard_1.1.3.1.7, WebBrowserStandard_2.1.2.3.1, and WebBrowserStandard_2.2.1.1.1 needs a Transformation describing pre- and post Caps sets.
E.g. for WebBrowser_2.2.1.1.1 there is the following:
a) It provides Caps 'WebBrowserStandard_2.2' being a super set including all Caps provided by WebBrowserStandard_2.1.2.3.1, but *not* all Caps provided by WebBrowser_1.1.3.1.7 (since the major number has changed);
b) It may require Caps ('SocketStuff_3.4', 'RenderingMachine_2'). A package RenderingMachine_2.1 would provide the last requirement.
c) It may conflict
- with package WebBrowserNicerButSlow in general (since it uses the
same sockets); and
- with SocketStuff_3.4.3.7.x (arbitrary x), since the bugfix there had
some very special side effect - unfortunately another installed important package PWorkaround needs exactly one of this versions of SocketStuff: it made a workaround not compatible with a newer version of it, where the side effect has been removed... (world is not perfect).
Note: if the user wants to install WebBrowserNicerButSlow or any other package, all packages have to be checked for conflicts.
How to model c)?
- The conflict with package WebBrowserNicerButSlow is an all over
package conflict and will be modeled as a conflict with Caps ('WebBrowserNicerButSlow'), which includes all Caps ('WebBrowserNicerButSlow_a.b.c.d.e') for arbitrary a, b, c, d, e.
- The conflict with SocketStuff in versions 3.4.3.7.x is *not* an all
over package conflict, it is a package-version conflict. This will be modeled as a conflict with Caps ('SocketStuff_3.4.3.7').
Ok, you differentiate with "all over package" conflicts and "package-version conflicts".
"All over package" also denoted "general package" or in this mail (#$% confusion).
d) It has a Transformation requires: ('SocketStuff_3.4', 'RenderingMachine_2.1') conflicts: ('SocketStuff_3.4.3.7') | V provides: ('WebBrowserStandard', "all over package Caps" 'WebBrowserStandard_2', "major release" 'WebBrowserStandard_2.2', "minor release" 'WebBrowserStandard_2.2.1', 'WebBrowserStandard_2.2.1.1', 'WebBrowserStandard_2.2.1.1.1', 'WebBrowser' "logical Caps"). The last Caps in the result is a logical Caps provided by both packages WebBrowserStandard and WebBrowserNicerButSlow. Such a beast makes sense for expressing dependencies which shouldn't be bounded to a certain package: in this example there may be a point in the system automatically starting the installed WebBrowser, whatever it may be (WebBrowserStandard or WebBrowserNicerButSlow).
The question in case of holding package PWorkaround is, if there is a version SocketStuff_3.a.b.c.x with a >= 4, b, c, x arbitrary, but not ((a = 4) & (b = 3) & (c = 7)), which also gives the input Caps 'SocketStuff_3.4'! Otherwise it wouldn't be possible to install WebBrowserStandard without removing PWorkaround first. So the package-version conflicts are reducing the number of packages capable of providing the needed input caps.
I am sorry Stephan but my brain didn't parse all this.
I hope now it has been easier.
I am fearing the complexity here.
It is easier as it seems, I think.
[SNIP of similar description of upgrades that also fried my brain :)]
[SNIP of make tool]
I've made some efforts to give examples: <rant> If you don't give examples, people want to have them. But if you give them, what they want, they don't like them... </rant>
To Lex's example
[SNIP of description of example]
Thinking in capability sets and a versioning policy is needed.
Well... I am not sure it is "needed", but that doesn't mean we can't prepare for it. My summary is at the end.
I'm holding my opinion of needing a good versioning scheme for a good automatically upgrading mechanism. This versioning scheme may correspond to categories as I've described above. The capability set thinking just allows to group the versions as needed for automatically upgrading.
But testing is nevertheless important: it would be very good to have many automatically runnable tests to classify package combinations as stable/unstable/bleedingEdge/etc.. Since we are faced with a combinatorial explosion here, hints regarding stable/unstable/bleedingEdge/etc. from the package maintainers would be helpful though...
The stable/unstable categories are today mandatory per package release. In other words - when you register a new release today you have to specify maturity level AND compatibility level (giving something similar to your version number scheme).
These categories are good for prefiltering of packages to be visible (or not) by the dependency resolver! In my humble thinking... ;-)
Relationships with existing tools and concepts ##############################################
SM
If the proposed version policy would be used for classifying packages at SM, the release flag of packages would be replaced or modelled by a corresponding version number, which is much more fine granular.
You are saying "release flag". I assume you mean the "published" flag (though I agree that it could have been called "release" or something). And I don't agree. :) That flag is meant to tell people that even though that release is available on SM it SHOULD NOT BE DEPENDED upon. The maintainer might just be testing it etc. It is a "hey, this release might not even be here - I may delete it at any second"-thing.
OK, bought your point. Another good prefiltering category.
And also - the compatibility level categories recently introduced seems very much like your version numbering scheme.
They are ;-)
An important question: Who is responsible for maintaining logical packages?
You mean "Caps"?
I mean additional Transformations providing logical Caps in addition to the standard Caps.
An - already known - example: packages WebBrowserStandard and WebBrowserNicerButSlow may both generate Caps 'WebBrowser' by an installation. There also may be a logical package WebBrowser which just depends on one of them. But who makes the choice? This is similar to the question, who decides which stuff is worth enough to come into the official distribution.
It would be interesting to enrich SM with a dependency mechanism as described in this paper...
Yes, at least in some regards - see my proposal/summary below.
[SNIPP of some stuff]
Roadmap (loosely, may change) #############################
- Posting this mail and looking for the reactions.
*** If there is enough interest: ***
Yes, sorry for taking a while to respond. It overwhelmed me at first. :)
I feel with you and experience hard work feelings while writing this reply, too... But I think it is worth it!
Creating a Swiki page with this stuff as a start.
Getting a consensus for a versioning policy by
- getting feedback via ML discussions,
- posting an improved draft,
for some iterations. To come to a consensus at this point is *very* important, *independently* from the chosen technical solution!
Yes - or at least we need some form of concensus regarding the structures in SM that can support different approaches.
This sounds promising. But to get a good automatically upgrading... you know my point.
One more: automatically upgrading also comprises installing a new general package needing some upgraded version and at the same time staying *compatible* with another installed package written for the not upgraded version.
- Going further with discussions about technical ideas: the concepts
should be good *before* coding. This doesn't mean to discuss forever and not to start with coding after a while...
Personal remarks ################
Don't try to be too perfect: KISS!
Yes. And I am sorry Stephan, but this model is... not really KISS in my book. :) But I like parts of it.
I have to admit that KISS seams to be quite complex in this domain...
Hehe. :)
Since the domain is complex, if you want to have a *good* result (automatically...) :)
I'm willingly to invest some more time - e.g. by coding - into Dependencies for Squeak, but there should be a good chance, that the results won't be thrown into the trash can...
We should try to ensure that.
Ok, here is my summary (phew):
- Caps is an extra "indirection layer" for expressing more abstract
dependencies. It should be possible to add "on top" of SM. It is a layer I will not work with myself, because I think it is an overkill. But that doesn't mean we can't make sure that the structures in SM allow for it to be added.
My suggestion here: please try to find arguments why the "overkill" isn't needed for a good automatically..., and how to get it with more modest concepts. Just trying to find them helps in understanding the domain better!
- Caps in themselves seems to be very easily modelled using
SMCategories. The category tree is currently centrally managed by me, in other words - by the Guides. I assume that would be the same for Caps, so it seems to fit.
This is interesting, since it could be a way for trying both approaches in parallel: just one category for the DEPS version number (or more along the mapping highlight described above, but this could be "overkill") would be sufficient to realize a DEPS prototype, I think.
The water in the wine: the install/deinstall hooks for general packages should have some well defined properties, I haven't tried to describe in greater detail so far. But this applies - again - to whatever *good* automatically ...
- Your version numbering scheme seems overly complex IMHO. I am not
sure if the current "Compatibility level"-as-categories-approach fulfills your needs there. Note that the actual sub categories representing each level can easily be changed - I just whipped up a few, see http://map1.squeakfoundation.org/sm/category/cb51b604-bb97-415d-a040-f23 1ecdd5bc7
Just taking a look... I'm surprised (seeing this page the first time, I think): this goes *** very close *** in direction of DEPS versioning scheme! OK, the representation varies, but this is no problem, if my mapping idea above holds.
Well, my head is spinning a bit, but what about this:
I have in my dev image added the concept of resources. A resource is like an "attachment" to another SMObject. This way other tools can attach metadata to for example SMPackages or SMPackageReleases that SM doesn't really know how to handle - but doesn't care about. At this point - since all modifications to the map must be done at the master server - this would only be doable using a remote call to the master server - but we can add those.
I then have a concrete subclass of SMResource called SMPackageReleaseConfiguration. It holds a list of required SMPackageReleases and a status flag. The flag can currently be #working or #failing. #working means that the release we attach the config to "works" with those specific required releases installed. It is thus a tested configuration, as I have described before.
Is a SMPackageRelease one specific version (marked as release or so) or a set of possible versions each fulfilling the same definition of belonging to the same release? I'm somewhat in the fog here.
We also already have the mandatory categorizations of releases "Compatibility level" and "Maturity level".
Now - let's say I continue with my "simpler" model that doesn't have Caps and focuses on the SMPackageReleaseConfigurations as the basis for figuring out "what do to". My mechanism is built in a separated SMDependencyEngine. We may even figure out a set of "services" that such an engine should be able to give.
One simply service would be "Ok, given that I want Seaside, Monticello an Shout in this image - what specific installs and upgrades should I perform and in which order?". This is btw the first service I am trying to implement as a spike solution.
So if we agree on a set of such services - or in fact different engines could offer different services and we can just let the engine hook itself into the menues in the package tools and offer "what it has" - then you can build your Caps-based engine on top of SM and use:
- Caps as categories. They would then be optional to use by the package
maintainers.
Should work. Just needed for additional Caps to the standard ones.
- Compatibility level (as exists today, mandatory)
Should work.
- Maturity level (as exists today, mandatory)
Good for prefiltering.
- SMPackageReleaseConfigurations (as I have described and that I am
adding)
I think I wouldn't need them, but I could need the anti-configurations you have recently mentioned, for expressing conflicts. Perhaps.
- SMResource (so that you can optionally add extra metadata to
SMObjects inside the map)
Always good to have this possibility: is there a 1-to-1 relationship between an SMRessoure and a package-version combo (installable entity)?
- And your engine can hook into the SqueakMap Package Loader menus when
installed.
This sounds promising!
Perhaps you are disappointed at me for not buying your solution - I hope not. :)
I'm not disappointed, since I - think this mail will help to get the different concepts clearer; - haven't seen any serious flaws in my concepts so far (but they may come later, of course); since we - have found a mapping between DEPS version numbers and categories; and you - seem to be open minded and - don't seem to fear a potential competitor (this is great!).
Caps is an interesting approach - even though I think it is overkill.
I don't think so ;-)
We share the idea of recording the compatibility level of releases -
Yes.
but I don't like encoding it in version numbers.
It's just an encoding: what do you think about the 'mapping highlight'?
But I desperately want SM to work as a "base" for testing different approaches here, so please - tell me how this sounds
This sounds good!
and what you think/need to consider moving forward with your idea.
I'd like to continue this discussion until I'm sure, - that you have understood my points, - that I have understood your points; with the goal to better model the domain for getting a good au... ;-)
Greetings Stephan
regards, Göran