Hi Göran and all!

Göran, thanks for your constructive reply.

After reading it once, I have the impression, that this could lead to an
interesting debate *improving* the concepts.

<Some time later>
I'm a little bit tired now... ;-)

I've sitted one night to write this answer and many things have become
clearer while thinking about and writing the answers followed by
iterated thinking about previously treated parts. I've been faced with
misunderstandings and hopefully my thoughts about your thoughts are
coming close to reality now...
So I suggest to read this to the end before hitting the reply button and
starting writing, many things should become clearer while going this
long way and if gotten the whole picture...

Göran, please don't feel forced to reply fast, I really understand if
this takes a while...


But now let's go!


At first I want to clearify some definitions to avoid confusion,
which has arosen on both sides, I think.

I've used the term Package (uppercase 'P') *always* in the narrower
sense of 'one package-version combo', e.g. Package WebBrowser_2.1.3.4.1.
There is admittedly at least one exception (the headerline 'Package
WebBrowser' below obviously denotes a bunch of versions, but it is just
'packages' written as section headerline with uppercase first letter).

And I've used the term 'package' context dependent: sometimes as
Package, but also sometimes more general, e.g. package WebBrowser. I
have found this more general use in *opposite* to your definition, which
seem to match the definition of DEPS Packages above, as I see it now (at
least these interpretations have led to a consistent picture in my head).

I think we have an overloading of the term 'package', which leads to
confusion.

Is 'package' for you a shortcut for a package-version combo, which would
be the same as a DEPS Package above?
Do you mean with 'release' one of many packages (package-version combo),
stating that something important has changed in the code, of some kind
of "Don't know how to avoid the word 'pack...'" doing something (e.g. a
WebBrowser "Don't know...")?


Another problem: I've also been confused by the term 'release': I would
expect to have only releases - tagged stable versions in a version
control sense - at SM, but you seem to have another
definition/policy/whatever there...


I hope this helps for reading the rest (there is at least one highlight
waiting for you :-) ):

goran.krampe at bluefish.se wrote:
> Hi Stephan and all!
> 
> I got a bit tired of writing long postings, but since I definitely don't
> want to loose the opportunity to cooperate with Stephan here I will take
> the time to reply to this one know.
> 

> Before I start - Stephan - don't be offended if I don't agree with you
> now. :)

No problem.

> 
> And I will try to snip parts that I have no comment on.

Good (I have asked for it).

> 
> 
> Stephan Rudlof <sr at evolgo.de> wrote:
> [SNIP]
> 
>>Technical Terms
>>###############
>>
>>Caps
>>====
>>There are Capabilities (Caps) each describing the capability to do
>>something.
>>Caps mostly depend on other Caps, which are prerequisites of
>>them.
>>In the following Caps means Capabilities or a union of them.
> 
> 
> This "Caps" layer is an extra abstraction layer that I am not sure
> about.
> I understand the theory behind it - but it feels a bit on the overly
> complex side,
> at least if it is the primary mechanism. If it was an additional
> abstraction that one could select to use or not, then it would be
> another matter.
> 
> But I will read on first... :)
> 
> 
>>Transformations
>>===============
>>
>>Transformations describe how it is possible to change a set of Caps:
>>they may - virtually at first - add or remove Caps or both. They
>>transform one set of Caps into another one. To do their work they
>>perform Actions.
>>
>>Inputs are
>>I.1. requires: required Caps, and
>>I.2. conflicts: conflicting Caps,
>>outputs are
>>O.1. provides: newly generated Caps, and
>>O.2. deletes: deleted Caps.
>>Properties:
>>- provides are const sets, but
>>- deletes are configuration dependent.
>>
>>Outputs are expressed as
>>  - set of (named) Caps.
>>
>>Inputs are expressed as
>>  1. set of (named) Caps (as for outputs), or
>>  2. as a Block
>>     - taking a Caps set as an argument, returning
>>       - true, if the requirement/conflict has been fullfilled,
>>       - false, otherwise.
> 
> 
> Hmmm, ok - at first look this seems again pretty... advanced. Or
> complex.
> 
> 
>>To Blocks as input
>>------------------
>>It is computational expensive to use Blocks as input, since in principle
>>all existing Caps and those somehow generatable by whatever
>>Transformation combination (!) have to be checked.
>>
>>An - important, I think - improvement for the case of expressing
>>conflicts may be to divide the Blocks into two classes (later there may
>>be more):
>>  1. Blocks just checking conflicts with Caps generated additionally by
>>the packages generating the requirement Caps, under the precondition,
>>that the requirement Caps are given as set.
>>  2. Other Blocks.
>>
>>See section "To Lex's example" below for a motivation of Blocks as
>>arguments here and the suggested improvement.
> 
> 
> Ok, this part isn't at all clear to me at this point. But it is only a
> reflection so...
> 
> 
>>Boolean logic
>>-------------
>>Require sets model a logical AND of requirements. Logical ORs - and
>>therefrom Boolean logic - can be modeled via additional Transformations
>>with newly introduced Caps names. This is far more efficient as trying
>>to do this in one Transformation using the Blocks mechanism!
> 
> 

> Ouch. Ok, my first reaction is "NO!". :) I know it is tempting to add
> the capability to use boolean logic to express dependencies/scenarios or
> whatever - and Debian uses it - but personally I have elected to NOT go
> there because the complexity seems to me to be unwarranted.

If I've understood you correctly below, you also have AND logic
(multiple requires) in your logic. A logical Caps just extends this to
also have an OR.

How would you model something like
  WebBrowserUsingApp requires:
    (WebBrowserStandard | WebBrowserNicerButSlow)
; without some kind of Boolean logic?

With Caps there are two ways to go in this example.

The first one assumes there is no logical Caps 'WebBrowser'. Then there
is a need to introduce two new - very simple - Transformations

requires:  ('WebBrowserStandard')
   |
   V
provides:  ('WebBrowser').

requires:  ('WebBrowserNicerButSlow')
   |
   V
provides:  ('WebBrowser').

and to require 'WebBrowser' in the 'WebBrowserUsingApp' generating
Transformation.

The second alternative is chosen below: there the idea of having a
logical 'WebBrowser' has been at 'package providing/deleting
Transformation' creation time of WebBrowserStandard and
WebBrowserNicerButSlow, so this Caps 'WebBrowser' has just been added to
their provides then.
Note: Puh, what a huge word combo is
  "at 'package providing/deleting Transformation' creation time"
! But I've wanted to be as precise as possible. I think we need some
definitions, but not now...

The good think here is, that it is possible to *add* such a logical
choice *afterwards*, if it hasn't been done at writing the
Transformations for one of the Browser packages, without the need to
change them! So later there can be better and better Transformation
rules, with different originators: there needn't be one central instance
(without such a mechanism you had to change the orginal Transformations
in direction of the second alternative.

Note: This example doesn't forbid to ask the user for a decision if the
dependeny resolver comes to this point. But this is an UI issue.


To the complexity issue: What is complex here?

I don't see any problems into traversing a graph of nested AND/OR
connected requires and trying the alternatives if there is an OR (the
AND exists in every non-trivial dependency logic, I assume).

Am I missing something?

> 
> But ok, shutting up and reading on... :)
> 
> 
>>Outlook to packages
>>-------------------
>>A package removal can be modeled by transforming the all over package
>>Caps as requires into deletes of this and all other package related Caps.
> 
> 
> Don't understand, but not important at this point I guess.
>  
> 
>>How to model dynamic provides?
>>------------------------------
>>The restriction 'provides are const sets' eases some things, but it is
>>not a limitation AFAICS.
>>
>>Example: Installing a package WebBrowser may result into more Caps if
>>some other package SuperFonts is there.
>>This can easily be modeled by a *logical* (no Action) package
>>Transformation describing the all over functionality of the packages
>>combo, with
>>  - requires: Caps ('WebBrowser', 'SuperFonts'), and
>>  - provides: something like ('WebBrowserWithSuperFonts').
> 
> 
> Ok, at this point I am thinking - perhaps this Caps layer should be
> something on top of SM?
> This may mean that your model could be a layer on top of mine. Just a
> feeling at this point.
>   
> 
>>Note
>>----
>>It's questionable, if naming both set and Block inputs the same is a
>>good idea, but I want to stop for now...
>>
>>
>>
>>Actions
>>=======
>>Actions are commands leading to a change of the (set of) Caps in the
>>system. If triggered, they add and/or remove one or more Caps in reality.
>>
>>
>>
>>Algorithms
>>############
>>
>>Before there is no good model of the domain, it is not urgent to think
>>about algorithms. I don't fear not to find some...
>>
>>A few things I'm currently expecting to be faced with:
>>Easy:
>>- depth first traversing of dependency graphs,
>>- back tracking with detection of cycles.
>>Not so easy:
>>- combinatorial explosion, if Blocks as inputs for Transformation
>>requires and conflicts come into play (but section 'Transformations->To
>>Blocks as input' contains a first idea for embanking it).
> 
> 
> Yes, the reason for me starting to write code was that I wanted to
> explore at least some of the algorithms involved.
> 
> 
>>Application to packages
>>#######################
>>
>>Packages
>>========
>>
>>A Package provides Caps: these are representing some things to be made
>>with or by the Package (interfaces, commands). There may be more than
>>one Caps per Package!
> 
> 

> Since this would be a thing changing over time I would presume that
> actually a package *release* provides Caps?

After my definitions:
- *Each* DEPS Package provides a *set* of Caps.
- Some sets of package-version combos (or Packages) are sharing the same
Caps (one or more).
- A set of package-version combos belonging to the same e.g. WebBrowser
general package shares one or more of the provided Caps! At least after
the proposed versioning number scheme below...

So a 'package *release*' - as I've understood you - has the following
properties in DEPS notation:
- it provides a set of Caps,
- it should have a change in one of the higher ranked version numbers
(to express the semantically important code change),
- it provides a Caps (in the set of its provided Caps) shared by a set
of Packages (all are providing it) containing this release and all
sub-releases (this could be used to express a dependency or conflict to
a set of Packages at release compatibility level).

I think this will become clearer if you read further to more versioning
number examples and - neat outlook - how to map them to categories and
vice-versa!

> 
> 
>>A Package needs Caps provided from other packages (requires).
> 
> 
> Same thing here. In my model this extra "redirection layer" is not
> present - there a release has tested configurations and each
> configuration more or less lists the other package *releases* that are
> needed for the release to work.

I see a problem here with your approach: you are faced with a
combinatorial explosion.
One package release requiring say 4 other packages each existing in say
4 releases leads to 4 ^ 4 = 256 configurations...

Let's assume, there are 10 tested configurations. Then you have to
switch to one of the 246 untested ones if there is only one other
general package requiring just one of the 4 requires in another release
(assumed that different versions cannot coexist).

To help me understand: Why aren't you seeing a problem here?

If I've understood you correctly, you would try to come through this via
the compatibility level categories combined with some rules for
controlled deviation from the tested configurations, but currently I
have serious stomaches here.


> 
> Which made me think perhaps this capability layer can be "on top" of my
> model.

More to this point below.

> 
> 
>>A Package providing some Caps may conflict with another Package
>>(providing other and/or the same Caps).
> 
> 
>>To be able to express these conflicts at Caps level it is necessary that
>>all packages are mutually different in at least one Caps! But this is
>>easy to accomplish, if there is just one Caps named so, that it contains
>>the - unique - name of them.
> 
> 
> Note that a Cap could easily be modelled using SMCategories. That is the
> point with SMCategory - it is a unique canonicalized "tag" that can be
> attached to SMObjects.

> I am not sure what you are saying though in the
> last part. Are you saying that two different packages (releases) can not
> provide the exact same Caps?

Exactly. And they don't after the proposed versioning scheme.

Some explanation, though: here Caps in the sense of union of Caps is
meant. E.g. general package Foo in version 4.4.3.2.3 provides standard
Caps (via a corresponding Transformation)
('Foo', 'Foo_4', 'Foo_4.4', 'Foo_4.4.3', 'Foo_4.4.3.2', 'Foo_4.4.3.2.3')
; another minimal changed version 4.4.3.2.4 of the same general package
provides standard Caps
('Foo', 'Foo_4', 'Foo_4.4', 'Foo_4.4.3', 'Foo_4.4.3.2', 'Foo_4.4.3.2.4')
. In this example both versions share all, but the last Caps.

Note: this representation is made from/for a semantic point of view; in
an implementation one #Foo_4.4.3.2.4 may be sufficient to represent all
standard Caps (but there may be more).

> Why not?

To be able to differentiate between two different package-version combos
at the Caps level.
This is needed to be able to express a conflict with just this special
seldom used one version of package Foo (e.g. expressed as conflict with
Caps 'Foo_4.4.3.2.3').

> 
> 
>>A conflicting Package has to be removed (this can be difficult) before
>>installing a new one, leading into a reduction of its provided Caps
>>(there may be logical ones left, which are also provided by another
>>package).
>>There may also be Packages, which only remove Caps without adding new
>>ones (system shrink)!
>>
>>
>>
>>Versions
>>========
>>
>>This section has become very long and is very crucial regarding its
>>social aspects.
>>
>>There *have* to be policies to be followed by the package maintainers.
>>I'm currently thinking of version numbers like
>>  1.1.3.4.2 #(1 1 3 4 2)
>>  2.7.3.1.1 #(2 7 3 1 1)
>>, where a change in one of the numbers means:
>>
>>1. incompatible, changed interface semantics, called major package
>>number: the package is expected to be incompatible with older or newer
>>ones having another number here (at this position);
>>
>>2. downwards compatible, called minor package number: the package has
>>changed, and their may be new features, but it is expected to work with
>>software written for lower numbers here (but with same 1. number, of
>>course);
>>
>>3. downwards *and* upwards compatible: the package has changed, but
>>there are *no* new features here (but it may be faster for example), so
>>it is expected to work with software written for higher numbers here
>>(but with same 1., 2. numbers, of course);
>>
>>4. compatible: 'small' fixes without changed functionality in the sense
>>above (also see 'Classification issues' below);
>>
>>5. no code changes at all: no changed semantics, but e.g. improved class
>>comments.
>>
>>After this scheme, a version number a.b.c.d. for a package P corresponds
>>to the Caps
>>  'P',
>>  'P_a',
>>  'P_a.b',
>>  'P_a.b.c',
>>  'P_a.b.c.d',
>>  'P_a.b.c.d.e'.
>>Caps 'P_a.b' - normally - includes *every* Caps 'P_a.x', x < b, since
>>the functionality/interface has just been smaller then. The longer
>>numbers are for expressing conflicts and requirements in a world not
>>being perfect.
>>This means, we finally have defined a relationship between Caps and
>>version numbers!
>>And it should be clear now, why there is this funny plural with the
>>'Caps' term: it is a normal case for a package-version combo, that it
>>provides a *set* of Capabilities!
> 
> 

> I am not comfortable with adding yet another version numbering scheme.
> It seems simply too complex to me.

I have more feared, that it could be too simple ;-)
As I've written in another Mail to Stan:
"The granularity of the proposed scheme is fine, but better too fine as
 too simple: nobody is forced to publish versions only changing in lower
 ranked version numbers, but it is possible. And it is also possible to
 be very fine granular in expressing conflicts with other package
 versions: from a singular version to larger and larger sets of versions."


I imagine one simple dialog asking a package release publisher five or
less questions for computing a new version number (for all increase
rules applies, don't change higher ranked version numbers, reset lower
ranked ones (similar to numbers ;-) )):

1. Do you expect, that
    most code running with the previous version,
    does *not* work anymore with this new version?
yes -> increase 1., finished; no -> continue...

2. Do you expect, that
    most code running with the previous version,
    continues to work with this new version?
yes -> increase 2., finished; no -> continue...

3. Do you expect, that
    most code to be written for running with this new version
    runs with the previous version, too?
yes -> increase 3., finished; no -> continue...

4. Do you expect, that
   3. applies
   *and* have you made a small fix without adding new functionality?
yes -> increase 4., finished; no -> continue...

5. Do you have *not* changed any code at all (just comments, etc.),
   so that's *impossible* that other code breaks?
yes -> increase 5., finished.

Of course with 'version' is meant a published/public version
  *** thought to be used by others ***
here, so these questions don't apply to 'private' versions (probably
they shouldn't go into the dependency system at all, but manually
installed).


It should be possible to answer these maximal 5 questions for a package
maintainer (beeing the target group here), otherwise I wouldn't like to
have his/her package... ;-)


> 
> SMPackageReleases has today one manually edited field "version" and one
> automatic version number which can't be edited.
> I want to express the "level of change" in one release compared to the
> previous release using categories instead. And I have created a first
> shot at these categories in SM and made it mandatory to select one of
> those when you register a new release.

If you'd choose the categories along the questions above, the DEPS
version numbers could be computed automatically (coming again as
highlight below)...

Some prospects for convergation here :-)

> 
> 
>>Classification issues
>>---------------------
>>It seems to be difficult to classify fixes (4.) correctly.
>>Fixes definitely change the formal semantics, so one has a good reason
>>to rank them higher and exchange them with 3. for example. A version
>>number change in 3. normally doesn't change the formal semantics (but
>>may introduce new bugs as a change in 4. as well, changing the formal
>>semantics then).
>>
>>But if a 'small' fix just makes the package working with some seldom
>>used other package being the only one running into the bug, then the
>>classification as 4. makes sense (3. may introduce new bugs affecting
>>many packages).
>>But if the fix removes some serious problems affecting many other
>>packages, it may as well classified as 2., since there is good reason to
>>view this as an important upgrade.
>>
>>A similar argumentation could be made for changes in the functionality
>>classified as 3.: a piece of code just running much faster without
>>changing the formal semantics, could open up new use cases. This may as
>>well earn a classification in 2..
>>
>>Currently I think the intelligence of the package maintainers is
>>questioned here, since 'it depends'...
> 
> 
> I interpret this to mean that the classification is not a black and
> white thing

Yes.

> - but that it helps a lot to have.
> I agree.

I have to think here.
Without this classification there would just be standard Caps like e.g.
for general package Foo in version 4.4.3.2.3:
  ('Foo', 'Foo_4_4_3_2_3')
('_' instead of '.' for not thinking they are made from real DEPS
versions), and you wouldn't have any semantically meaningful
relationship of the latter (version) Caps
  'Foo_4_4_3_2_3' to Caps
  'Foo_4_4_3_2_4' of the next version.
So
  'Foo_4_4_3_2_4' is equally 'far' away from
  'Foo_4_4_3_2_3' than e.g.
  'Foo_9_9_9_9_9'.

Automatically upgrading wouldn't work; so classification is *needed*
here. The classification may be wrong, but this can be repaired via
more/bigger Transformations and/or prefiltering (e.g. by disabling
packages with a compatibility classification too optimistic (too
pessimistic doesn't break anything)) of packages to be considered for
Transformations at all.

> 
> 
>>Note to fixes
>>-------------
>>An important fix (changed version number 4.) can make the package work
>>the first time with another one. It may make sense - especially if the
>>major number recently has changed - to apply such a fix to an older
>>version, too!
>>This would lead to a new older package with a new number, of course.
>>
>>
>>Automatically upgrading
>>-----------------------
>>In general automatically upgrading should
>>- apply to all packages increasing their minor or lower ranked version
>>number,
>>- *not* apply to packages increasing their major version number.
>>
>>A harvesting process could be introduced using this technical solution
>>as follows: addditional classifications of packages as
>>stable/unstable/etc. could prefilter the number of packages to be
>>searched for fulfilling dependencies (this classification could also be
>>realised by just putting them into classified - different - package
>>directories).
> 
> 

> I think exactly how auto upgrading will work will be heavily
> parameterized

Agreed.

> and also ask the user for input.

Agreed, but with limitation:
An end user just using Squeak only wants to have a successful
installation with as less questions as possible (and the end user often
wouldn't know how to answer).

> And different people can have different engines

Yes. But there will be some default.

> - so this isn't crucial
> stuff to "agree on".

Please don't be so fast here:
- as better the classifications, as better the automatically upgrading;
- as worse the classifications, as worse the automatically upgrading!

This won't be changed by another parametrization or engine.
Or did I miss a point here?

> 
> 
> 
>>Why major version number 1. at all?
>>-----------------------------------
>>After this versioning policy automatically upgrading would (mostly) work
>>from version to version with a *fixed* first version number. If the
>>major number changes, there are incompatible versions (since we have
>>defined it so!). This could lead to the question, why not to ommit this
>>major number and to force the package maintainer to publish a new
>>package with a different name instead.
>>
>>Arguments for a major version number increase instead of a name change:
>>
>>- There are cases for expressing the existence of some functionality at
>>a very high level independent from the major number: e.g. some package
>>wants to have a package WebBrowserStandard for enforcing to give the
>>user a chance of viewing its generated HTML files (without assuming any
>>API for calling it itself): a newly created name (how to guess it for
>>other package maintainers?) instead of a major number increase would
>>make expressing this dependency impossible.
>>
>>- Though an interface has changed, the functionality may be very
>>similar, and the package name should express functionality.
>>
>>- The package name combined with the major number could be seen as the -
>>technically and socially - needed new name.
> 
> 
> 
> Personally I think the identity of a package should foremost be the
> identity of the code base evolving.

> Sure, people can always elect to "fork" a package, by creating a new one
> - but hopefully in the coming SM that supports branches in the release
> tree such forks will be even more rare.

Increasing 1. can be considered as starting a fork (after the proposed
DEPS versioning scheme).

While thinking about this there has come the following question into my
mind: what is the difference between a
  versioning scheme needed for
    a package dependency system with *automatically* upgrading, and a
  versioning scheme needed for
    a version control system?

The main difference - as I currently see it - is, that
- for the package dependency system, versioning should reflect the
compatibility *categorization/classification*, which is a pure
semantical thing; and
- for the version control system, versioning just says, how far two
versions are *technically* away (e.g. by applying patches), which as
itself says *nothing* about the semantical changes.

> 
> 
> 
>>Consequences
>>------------
>>
>>This is a client POV versioning policy.
>>There may be others, of course: IMHO it's not so important which policy
>>to have, but
>>
>>    *** It's very important to have a policy at all! ***
>>
>>Otherwise a technically working dependency mechanism doesn't work for
>>packages because of social reasons...
>>
>>Technically this means to define some
>>  Caps <-> version number
>>relation with an order of Caps corresponding to an order of
>>version numbers like described above (or to use a similar mechanism).
>>
>>    *** Otherwise the technical solution *cannot* work! ***
> 
> 

> But I assume we can still represent the "level of change" using
> categories instead of version numbers?

Here comes the promised *highlight* a little bit more detailed:

If each package-version combo gets categories corresponding to answers
to the questions to package maintainers as written above: then we even
have a *mapping* from categories to version numbers (incrementally
computed) and vice versa (direct mapping).

Isn't that nice?

> 
> 
>>Installation of packages
>>========================
>>In general there are
>>- pre-install,
>>- do-install,
>>- post-install,
>>- pre-remove,
>>- do-remove,
>>- post-remove,
>>Scripts (they may be empty). They are triggered by performing an
>>(package install/removal) Action.
>>
>>Normally the install phase results into adding all provided Caps and the
>>remove phase into the removal of them. Potential conflicts have to be
>>solved *before* installation!
> 
> 
> Personally I think we should strive to get away from scripts - they just
> break. :)

> Seriously.I would rather try to use a declarative approach as long as
> possible and then only add hooks for scripts if we find out it is really
> needed.

I didn't want to give a definition for 'script' here; this part has
mainly been written down to show, where I see some reasonable hooks,
whatever 'scripts' should be. Better had been to replace
  "Scripts (they may be empty)."
by
  "Scripts (they often are empty)."

> 
> Scripts can also seriously trip us up and produce weird results that we
> will have a hard time tracking down.

Yes, of course.

>  
> 
>>Different kind of installs
>>--------------------------
>>There are different kind of installs:
>>- upgrading,
>>- full installation with a previous removal of an older version (if it
>>exists).
> 
> 
> Yes, currently SM actually differentiates these but only MC has the
> capability of doing things "differently" when upgrading - and it figures
> that out on its own today.
> 
> 
>>Examples (for looking, if the concepts are working)
>>##################################################
>>
>>Package WebBrowserStandard
>>==========================

'Package': normally I've meant the package-version combo, here the more
general 'package' has just been written as headerline...

>>
>>- Provides Caps 'WebBrowser' and more;
>>- needs Caps 'SocketStuff' and 'RenderingMachine';
>>- exists in versions 1.1.3.1.7, 2.1.2.3.1, 2.2.1.1.1;
>>- conflicts with package 'WebBrowserNicerButSlow' and some special
>>SocketStuff versions.
> 
> 
> I note that you have the Caps on the package and not on the releases.
> This would mean they can't change over time, or rather - the releases
> can't - at a specific point in time - have different Caps.

They can!
Via changed and/or newly added Transformations.
Caps besides the standard Caps (since they stay forever) could be
changed arbitrarily.

> 
> I also note you are expressing conflicts on package level, which also
> suffers from the same problem.

For me one package-version combo (hopefully the same as exactly one
installable 'package') is the finest granular entity. It has a *set* of
standard Caps; e.g. (not in ST notation)

('Foo', 'Foo_4', 'Foo_4.4', 'Foo_4.4.3', 'Foo_4.4.3.2', 'Foo_4.4.3.2.3')

You can express conflicts to just this one package Foo_4.4.3.2.3 via a
conflict with (one of the standard) Caps 'Foo_4.4.3.2.3' only provided
by this package. But it is also possible to express a conflict to some
set of packages, e.g. by a stating a conflict with Caps 'Foo_4.4' many
versions are meant, exactly those all providing this Caps (e.g. packages
Foo_4.4.7.3.3, Foo_4.4.7.3.3).

> 
> 
>>Each package version combo
>>  WebBrowserStandard_1.1.3.1.7,
>>  WebBrowserStandard_2.1.2.3.1, and
>>  WebBrowserStandard_2.2.1.1.1
>>needs a Transformation describing pre- and post Caps sets.
>>
>>
>>E.g. for WebBrowser_2.2.1.1.1 there is the following:
>>
>>a) It provides
>>Caps 'WebBrowserStandard_2.2' being a super set including all Caps
>>provided by WebBrowserStandard_2.1.2.3.1, but *not* all Caps provided
>>by WebBrowser_1.1.3.1.7 (since the major number has changed);
>>
>>b) It may require
>>Caps
>>  ('SocketStuff_3.4', 'RenderingMachine_2').
>>A package RenderingMachine_2.1 would provide the last requirement.
>>
>>c) It may conflict
>>  - with package WebBrowserNicerButSlow in general (since it uses the
>>same sockets); and
>>  - with SocketStuff_3.4.3.7.x (arbitrary x), since the bugfix there had
>>some very special side effect - unfortunately another installed
>>important package PWorkaround needs exactly one of this versions of
>>SocketStuff: it made a workaround not compatible with a newer version of
>>it, where the side effect has been removed... (world is not perfect).
>>
>>Note: if the user wants to install WebBrowserNicerButSlow or any other
>>package, all packages have to be checked for conflicts.
>>
>>How to model c)?
>>- The conflict with package WebBrowserNicerButSlow is an all over
>>package conflict and will be modeled as a conflict with Caps
>>('WebBrowserNicerButSlow'), which includes all Caps
>>('WebBrowserNicerButSlow_a.b.c.d.e') for arbitrary a, b, c, d, e.
>>- The conflict with SocketStuff in versions 3.4.3.7.x is *not* an all
>>over package conflict, it is a package-version conflict. This will be
>>modeled as a conflict with Caps ('SocketStuff_3.4.3.7').
> 
> 
> Ok, you differentiate with "all over package" conflicts and
> "package-version conflicts".

"All over package" also denoted "general package" or in this mail (#$%
confusion).

> 
> 
>>d) It has a Transformation
>>requires:  ('SocketStuff_3.4', 'RenderingMachine_2.1')
>>conflicts: ('SocketStuff_3.4.3.7')
>>   |
>>   V
>>provides:  ('WebBrowserStandard',     "all over package Caps"
>>            'WebBrowserStandard_2',   "major release"
>>            'WebBrowserStandard_2.2', "minor release"
>>            'WebBrowserStandard_2.2.1',
>>            'WebBrowserStandard_2.2.1.1',
>>            'WebBrowserStandard_2.2.1.1.1',
>>            'WebBrowser'              "logical Caps").
>>The last Caps in the result is a logical Caps provided by both packages
>>WebBrowserStandard and WebBrowserNicerButSlow. Such a beast makes sense
>>for expressing dependencies which shouldn't be bounded to a certain
>>package: in this example there may be a point in the system
>>automatically starting the installed WebBrowser, whatever it may be
>>(WebBrowserStandard or WebBrowserNicerButSlow).
>>
>>The question in case of holding package PWorkaround is, if there is a
>>version SocketStuff_3.a.b.c.x with
>>  a >= 4,
>>  b, c, x arbitrary, but
>>  not ((a = 4) & (b = 3) & (c = 7)),
>>which also gives the input Caps 'SocketStuff_3.4'! Otherwise it wouldn't
>>be possible to install WebBrowserStandard without removing PWorkaround
>>first.
>>So the package-version conflicts are reducing the number of packages
>>capable of providing the needed input caps.
> 
> 
> I am sorry Stephan but my brain didn't parse all this.

I hope now it has been easier.

> I am fearing the
> complexity here.

It is easier as it seems, I think.

>  
> [SNIP of similar description of upgrades that also fried my brain :)]
> 
> [SNIP of make tool]

I've made some efforts to give examples:
<rant>
If you don't give examples, people want to have them.
But if you give them, what they want, they don't like them...
</rant>

> 
> 
>>To Lex's example
>>================
>>
> 
> [SNIP of description of example]
> 
>>Thinking in capability sets and a versioning policy is needed.
> 
> 
> Well... I am not sure it is "needed", but that doesn't mean we can't
> prepare for it. My summary is at the end.

I'm holding my opinion of needing a good versioning scheme for a good
automatically upgrading mechanism. This versioning scheme may correspond
to categories as I've described above.
The capability set thinking just allows to group the versions as needed
for automatically upgrading.

> 
> 
>>But testing is nevertheless important: it would be very good to have
>>many automatically runnable tests to classify package combinations as
>>stable/unstable/bleedingEdge/etc.. Since we are faced with a
>>combinatorial explosion here, hints regarding
>>stable/unstable/bleedingEdge/etc. from the package maintainers would be
>>helpful though...
> 
> 

> The stable/unstable categories are today mandatory per package release.
> In other words - when you register a new release today you have to
> specify maturity level AND compatibility level (giving something similar
> to your version number scheme).

These categories are good for prefiltering of packages to be visible (or
not) by the dependency resolver! In my humble thinking... ;-)

> 
> 
> 
>>Relationships with existing tools and concepts
>>##############################################
>>
>>SM
>>==
>>
>>If the proposed version policy would be used for classifying packages at
>>SM, the release flag of packages would be replaced or modelled by a
>>corresponding version number, which is much more fine granular.
> 
> 
> You are saying "release flag". I assume you mean the "published" flag
> (though I agree that it could have been called "release" or something).
> And I don't agree. :) That flag is meant to tell people that even though
> that release is available on SM it SHOULD NOT BE DEPENDED upon. The
> maintainer might just be testing it etc. It is a "hey, this release
> might not even be here - I may delete it at any second"-thing.

OK, bought your point. Another good prefiltering category.

> 
> And also - the compatibility level categories recently introduced seems
> very much like your version numbering scheme.

They are ;-)

> 
> 
>>An important question:
>>  Who is responsible for maintaining logical packages?
> 
> 
> You mean "Caps"?

I mean additional Transformations providing logical Caps in addition to
the standard Caps.

> 
> 
>>An - already known - example: packages WebBrowserStandard and
>>WebBrowserNicerButSlow may both generate Caps 'WebBrowser' by an
>>installation. There also may be a logical package WebBrowser which just
>>depends on one of them. But who makes the choice? This is similar to the
>>question, who decides which stuff is worth enough to come into the
>>official distribution.
>> 
>>It would be interesting to enrich SM with a dependency mechanism as
>>described in this paper...
> 
> 
> Yes, at least in some regards - see my proposal/summary below.
> 
> [SNIPP of some stuff]
> 
> 
>>Roadmap (loosely, may change)
>>#############################
>>
>>0. Posting this mail and looking for the reactions.
>>
>>*** If there is enough interest: ***
> 
> 

> Yes, sorry for taking a while to respond. It overwhelmed me at first. :)

I feel with you and experience hard work feelings while writing this
reply, too...
But I think it is worth it!

>  
> 
>>1. Creating a Swiki page with this stuff as a start.
>>
>>2. Getting a consensus for a versioning policy by
>>  - getting feedback via ML discussions,
>>  - posting an improved draft,
>>  for some iterations.
>>To come to a consensus at this point is *very* important,
>>*independently* from the chosen technical solution!
> 
> 

> Yes - or at least we need some form of concensus regarding the
> structures in SM that can support different approaches.

This sounds promising.
But to get a good automatically upgrading... you know my point.

One more: automatically upgrading also comprises installing a new
general package needing some upgraded version and at the same time
staying *compatible* with another installed package written for the not
upgraded version.

> 
> 
>>3. Going further with discussions about technical ideas: the concepts
>>should be good *before* coding. This doesn't mean to discuss forever and
>>not to start with coding after a while...
>>
>>
>>
>>Personal remarks
>>################
>>
>>Don't try to be too perfect: KISS!
> 
> 
> Yes. And I am sorry Stephan, but this model is... not really KISS in my
> book. :) But I like parts of it.
> 
> 
>>I have to admit that KISS seams to be quite complex in this domain...
> 
> 
> Hehe. :)

Since the domain is complex, if you want to have a *good* result
(automatically...) :)

> 
> 
>>I'm willingly to invest some more time - e.g. by coding - into
>>Dependencies for Squeak, but there should be a good chance, that
>>the results won't be thrown into the trash can...
> 
> 
> We should try to ensure that.
> 
> Ok, here is my summary (phew):
> 

> 1. Caps is an extra "indirection layer" for expressing more abstract
> dependencies. It should be possible to add "on top" of SM. It is a layer
> I will not work with myself, because I think it is an overkill. But that
> doesn't mean we can't make sure that the structures in SM allow for it
> to be added.

My suggestion here: please try to find arguments why the "overkill"
isn't needed for a good automatically..., and how to get it with more
modest concepts. Just trying to find them helps in understanding the
domain better!

> 
> 2. Caps in themselves seems to be very easily modelled using
> SMCategories. The category tree is currently centrally managed by me, in
> other words - by the Guides. I assume that would be the same for Caps,
> so it seems to fit.

This is interesting, since it could be a way for trying both approaches
in parallel: just one category for the DEPS version number (or more
along the mapping highlight described above, but this could be
"overkill") would be sufficient to realize a DEPS prototype, I think.

The water in the wine: the install/deinstall hooks for general packages
should have some well defined properties, I haven't tried to describe in
greater detail so far. But this applies - again - to whatever *good*
automatically ...

> 
> 3. Your version numbering scheme seems overly complex IMHO. I am not
> sure if the current "Compatibility level"-as-categories-approach
> fulfills your needs there. Note that the actual sub categories
> representing each level can easily be changed - I just whipped up a few,
> see
> http://map1.squeakfoundation.org/sm/category/cb51b604-bb97-415d-a040-f23
> 1ecdd5bc7

Just taking a look...
I'm surprised (seeing this page the first time, I think): this goes
  *** very close ***
in direction of DEPS versioning scheme! OK, the representation varies,
but this is no problem, if my mapping idea above holds.

> 
> Well, my head is spinning a bit, but what about this:
> 
> I have in my dev image added the concept of resources. A resource is
> like an "attachment" to another SMObject. This way other tools can
> attach metadata to for example SMPackages or SMPackageReleases that SM
> doesn't really know how to handle - but doesn't care about. At this
> point - since all modifications to the map must be done at the master
> server - this would only be doable using a remote call to the master
> server - but we can add those.
>

> I then have a concrete subclass of SMResource called
> SMPackageReleaseConfiguration. It holds a list of required
> SMPackageReleases and a status flag. The flag can currently be #working
> or #failing. #working means that the release we attach the config to
> "works" with those specific required releases installed. It is thus a
> tested configuration, as I have described before.

Is a SMPackageRelease one specific version (marked as release or so) or
a set of possible versions each fulfilling the same definition of
belonging to the same release? I'm somewhat in the fog here.

> 
> We also already have the mandatory categorizations of releases
> "Compatibility level" and "Maturity level".
> 
> Now - let's say I continue with my "simpler" model that doesn't have
> Caps and focuses on the SMPackageReleaseConfigurations as the basis for
> figuring out "what do to". My mechanism is built in a separated
> SMDependencyEngine. We may even figure out a set of "services" that such
> an engine should be able to give.
> 
> One simply service would be "Ok, given that I want Seaside, Monticello
> an Shout in this image - what specific installs and upgrades should I
> perform and in which order?". This is btw the first service I am trying
> to implement as a spike solution.
> 
> So if we agree on a set of such services - or in fact different engines
> could offer different services and we can just let the engine hook
> itself into the menues in the package tools and offer "what it has" -
> then you can build your Caps-based engine on top of SM and use:
> 

> 	- Caps as categories. They would then be optional to use by the package
> maintainers.

Should work. Just needed for additional Caps to the standard ones.

> 	- Compatibility level (as exists today, mandatory)

Should work.

> 	- Maturity level (as exists today, mandatory)

Good for prefiltering.

> 	- SMPackageReleaseConfigurations (as I have described and that I am
> adding)

I think I wouldn't need them, but I could need the anti-configurations
you have recently mentioned, for expressing conflicts. Perhaps.

> 	- SMResource (so that you can optionally add extra metadata to
> SMObjects inside the map)

Always good to have this possibility: is there a 1-to-1 relationship
between an SMRessoure and a package-version combo (installable entity)?

> 	- And your engine can hook into the SqueakMap Package Loader menus when
> installed.

This sounds promising!

> 
> Perhaps you are disappointed at me for not buying your solution - I hope
> not. :)

I'm not disappointed,
since I
- think this mail will help to get the different concepts clearer;
- haven't seen any serious flaws in my concepts so far (but they may
come later, of course);
since we
- have found a mapping between DEPS version numbers and categories;
and you
- seem to be open minded and
- don't seem to fear a potential competitor (this is great!).

> Caps is an interesting approach - even though I think it is
> overkill.

I don't think so ;-)

> We share the idea of recording the compatibility level of
> releases -

Yes.

> but I don't like encoding it in version numbers.

It's just an encoding: what do you think about the 'mapping highlight'?

> But I
> desperately want SM to work as a "base" for testing different approaches
> here, so please - tell me how this sounds

This sounds good!

> and what you think/need to
> consider moving forward with your idea.

I'd like to continue this discussion until I'm sure,
- that you have understood my points,
- that I have understood your points;
with the goal to better model the domain for getting a good au... ;-)


Greetings
Stephan

> 
> regards, Göran
> 

-- 
Stephan Rudlof (sr at evolgo.de)
   "Genius doesn't work on an assembly line basis.
    You can't simply say, 'Today I will be brilliant.'"
    -- Kirk, "The Ultimate Computer", stardate 4731.3