Hi Stephan and all!

I got a bit tired of writing long postings, but since I definitely don't
want to loose the opportunity to cooperate with Stephan here I will take
the time to reply to this one know.

Before I start - Stephan - don't be offended if I don't agree with you
now. :)

And I will try to snip parts that I have no comment on.


Stephan Rudlof <sr at evolgo.de> wrote:
[SNIP]
> Technical Terms
> ###############
> 
> Caps
> ====
> There are Capabilities (Caps) each describing the capability to do
> something.
> Caps mostly depend on other Caps, which are prerequisites of
> them.
> In the following Caps means Capabilities or a union of them.

This "Caps" layer is an extra abstraction layer that I am not sure
about.
I understand the theory behind it - but it feels a bit on the overly
complex side,
at least if it is the primary mechanism. If it was an additional
abstraction that one could select to use or not, then it would be
another matter.

But I will read on first... :)

> Transformations
> ===============
> 
> Transformations describe how it is possible to change a set of Caps:
> they may - virtually at first - add or remove Caps or both. They
> transform one set of Caps into another one. To do their work they
> perform Actions.
> 
> Inputs are
> I.1. requires: required Caps, and
> I.2. conflicts: conflicting Caps,
> outputs are
> O.1. provides: newly generated Caps, and
> O.2. deletes: deleted Caps.
> Properties:
> - provides are const sets, but
> - deletes are configuration dependent.
> 
> Outputs are expressed as
>   - set of (named) Caps.
> 
> Inputs are expressed as
>   1. set of (named) Caps (as for outputs), or
>   2. as a Block
>      - taking a Caps set as an argument, returning
>        - true, if the requirement/conflict has been fullfilled,
>        - false, otherwise.

Hmmm, ok - at first look this seems again pretty... advanced. Or
complex.

> To Blocks as input
> ------------------
> It is computational expensive to use Blocks as input, since in principle
> all existing Caps and those somehow generatable by whatever
> Transformation combination (!) have to be checked.
> 
> An - important, I think - improvement for the case of expressing
> conflicts may be to divide the Blocks into two classes (later there may
> be more):
>   1. Blocks just checking conflicts with Caps generated additionally by
> the packages generating the requirement Caps, under the precondition,
> that the requirement Caps are given as set.
>   2. Other Blocks.
> 
> See section "To Lex's example" below for a motivation of Blocks as
> arguments here and the suggested improvement.

Ok, this part isn't at all clear to me at this point. But it is only a
reflection so...

> Boolean logic
> -------------
> Require sets model a logical AND of requirements. Logical ORs - and
> therefrom Boolean logic - can be modeled via additional Transformations
> with newly introduced Caps names. This is far more efficient as trying
> to do this in one Transformation using the Blocks mechanism!

Ouch. Ok, my first reaction is "NO!". :) I know it is tempting to add
the capability to use boolean logic to express dependencies/scenarios or
whatever - and Debian uses it - but personally I have elected to NOT go
there because the complexity seems to me to be unwarranted.

But ok, shutting up and reading on... :)

> Outlook to packages
> -------------------
> A package removal can be modeled by transforming the all over package
> Caps as requires into deletes of this and all other package related Caps.

Don't understand, but not important at this point I guess.
 
> How to model dynamic provides?
> ------------------------------
> The restriction 'provides are const sets' eases some things, but it is
> not a limitation AFAICS.
> 
> Example: Installing a package WebBrowser may result into more Caps if
> some other package SuperFonts is there.
> This can easily be modeled by a *logical* (no Action) package
> Transformation describing the all over functionality of the packages
> combo, with
>   - requires: Caps ('WebBrowser', 'SuperFonts'), and
>   - provides: something like ('WebBrowserWithSuperFonts').

Ok, at this point I am thinking - perhaps this Caps layer should be
something on top of SM?
This may mean that your model could be a layer on top of mine. Just a
feeling at this point.
  
> Note
> ----
> It's questionable, if naming both set and Block inputs the same is a
> good idea, but I want to stop for now...
> 
> 
> 
> Actions
> =======
> Actions are commands leading to a change of the (set of) Caps in the
> system. If triggered, they add and/or remove one or more Caps in reality.
> 
> 
> 
> Algorithms
> ############
> 
> Before there is no good model of the domain, it is not urgent to think
> about algorithms. I don't fear not to find some...
> 
> A few things I'm currently expecting to be faced with:
> Easy:
> - depth first traversing of dependency graphs,
> - back tracking with detection of cycles.
> Not so easy:
> - combinatorial explosion, if Blocks as inputs for Transformation
> requires and conflicts come into play (but section 'Transformations->To
> Blocks as input' contains a first idea for embanking it).

Yes, the reason for me starting to write code was that I wanted to
explore at least some of the algorithms involved.

> Application to packages
> #######################
> 
> Packages
> ========
> 
> A Package provides Caps: these are representing some things to be made
> with or by the Package (interfaces, commands). There may be more than
> one Caps per Package!

Since this would be a thing changing over time I would presume that
actually a package *release* provides Caps?

> A Package needs Caps provided from other packages (requires).

Same thing here. In my model this extra "redirection layer" is not
present - there a release has tested configurations and each
configuration more or less lists the other package *releases* that are
needed for the release to work.

Which made me think perhaps this capability layer can be "on top" of my
model.

> A Package providing some Caps may conflict with another Package
> (providing other and/or the same Caps).

> To be able to express these conflicts at Caps level it is necessary that
> all packages are mutually different in at least one Caps! But this is
> easy to accomplish, if there is just one Caps named so, that it contains
> the - unique - name of them.

Note that a Cap could easily be modelled using SMCategories. That is the
point with SMCategory - it is a unique canonicalized "tag" that can be
attached to SMObjects. I am not sure what you are saying though in the
last part. Are you saying that two different packages (releases) can not
provide the exact same Caps? Why not?

> A conflicting Package has to be removed (this can be difficult) before
> installing a new one, leading into a reduction of its provided Caps
> (there may be logical ones left, which are also provided by another
> package).
> There may also be Packages, which only remove Caps without adding new
> ones (system shrink)!
> 
> 
> 
> Versions
> ========
> 
> This section has become very long and is very crucial regarding its
> social aspects.
> 
> There *have* to be policies to be followed by the package maintainers.
> I'm currently thinking of version numbers like
>   1.1.3.4.2 #(1 1 3 4 2)
>   2.7.3.1.1 #(2 7 3 1 1)
> , where a change in one of the numbers means:
> 
> 1. incompatible, changed interface semantics, called major package
> number: the package is expected to be incompatible with older or newer
> ones having another number here (at this position);
> 
> 2. downwards compatible, called minor package number: the package has
> changed, and their may be new features, but it is expected to work with
> software written for lower numbers here (but with same 1. number, of
> course);
> 
> 3. downwards *and* upwards compatible: the package has changed, but
> there are *no* new features here (but it may be faster for example), so
> it is expected to work with software written for higher numbers here
> (but with same 1., 2. numbers, of course);
> 
> 4. compatible: 'small' fixes without changed functionality in the sense
> above (also see 'Classification issues' below);
> 
> 5. no code changes at all: no changed semantics, but e.g. improved class
> comments.
> 
> After this scheme, a version number a.b.c.d. for a package P corresponds
> to the Caps
>   'P',
>   'P_a',
>   'P_a.b',
>   'P_a.b.c',
>   'P_a.b.c.d',
>   'P_a.b.c.d.e'.
> Caps 'P_a.b' - normally - includes *every* Caps 'P_a.x', x < b, since
> the functionality/interface has just been smaller then. The longer
> numbers are for expressing conflicts and requirements in a world not
> being perfect.
> This means, we finally have defined a relationship between Caps and
> version numbers!
> And it should be clear now, why there is this funny plural with the
> 'Caps' term: it is a normal case for a package-version combo, that it
> provides a *set* of Capabilities!

I am not comfortable with adding yet another version numbering scheme.
It seems simply too complex to me.

SMPackageReleases has today one manually edited field "version" and one
automatic version number which can't be edited.
I want to express the "level of change" in one release compared to the
previous release using categories instead. And I have created a first
shot at these categories in SM and made it mandatory to select one of
those when you register a new release.

> Classification issues
> ---------------------
> It seems to be difficult to classify fixes (4.) correctly.
> Fixes definitely change the formal semantics, so one has a good reason
> to rank them higher and exchange them with 3. for example. A version
> number change in 3. normally doesn't change the formal semantics (but
> may introduce new bugs as a change in 4. as well, changing the formal
> semantics then).
> 
> But if a 'small' fix just makes the package working with some seldom
> used other package being the only one running into the bug, then the
> classification as 4. makes sense (3. may introduce new bugs affecting
> many packages).
> But if the fix removes some serious problems affecting many other
> packages, it may as well classified as 2., since there is good reason to
> view this as an important upgrade.
> 
> A similar argumentation could be made for changes in the functionality
> classified as 3.: a piece of code just running much faster without
> changing the formal semantics, could open up new use cases. This may as
> well earn a classification in 2..
> 
> Currently I think the intelligence of the package maintainers is
> questioned here, since 'it depends'...

I interpret this to mean that the classification is not a black and
white thing - but that it helps a lot to have.
I agree.

> Note to fixes
> -------------
> An important fix (changed version number 4.) can make the package work
> the first time with another one. It may make sense - especially if the
> major number recently has changed - to apply such a fix to an older
> version, too!
> This would lead to a new older package with a new number, of course.
> 
> 
> Automatically upgrading
> -----------------------
> In general automatically upgrading should
> - apply to all packages increasing their minor or lower ranked version
> number,
> - *not* apply to packages increasing their major version number.
> 
> A harvesting process could be introduced using this technical solution
> as follows: addditional classifications of packages as
> stable/unstable/etc. could prefilter the number of packages to be
> searched for fulfilling dependencies (this classification could also be
> realised by just putting them into classified - different - package
> directories).

I think exactly how auto upgrading will work will be heavily
parameterized and also ask the user for input.
And different people can have different engines - so this isn't crucial
stuff to "agree on".


> Why major version number 1. at all?
> -----------------------------------
> After this versioning policy automatically upgrading would (mostly) work
> from version to version with a *fixed* first version number. If the
> major number changes, there are incompatible versions (since we have
> defined it so!). This could lead to the question, why not to ommit this
> major number and to force the package maintainer to publish a new
> package with a different name instead.
> 
> Arguments for a major version number increase instead of a name change:
> 
> - There are cases for expressing the existence of some functionality at
> a very high level independent from the major number: e.g. some package
> wants to have a package WebBrowserStandard for enforcing to give the
> user a chance of viewing its generated HTML files (without assuming any
> API for calling it itself): a newly created name (how to guess it for
> other package maintainers?) instead of a major number increase would
> make expressing this dependency impossible.
> 
> - Though an interface has changed, the functionality may be very
> similar, and the package name should express functionality.
> 
> - The package name combined with the major number could be seen as the -
> technically and socially - needed new name.


Personally I think the identity of a package should foremost be the
identity of the code base evolving.
Sure, people can always elect to "fork" a package, by creating a new one
- but hopefully in the coming SM that supports branches in the release
tree such forks will be even more rare.


> Consequences
> ------------
> 
> This is a client POV versioning policy.
> There may be others, of course: IMHO it's not so important which policy
> to have, but
> 
>     *** It's very important to have a policy at all! ***
> 
> Otherwise a technically working dependency mechanism doesn't work for
> packages because of social reasons...
> 
> Technically this means to define some
>   Caps <-> version number
> relation with an order of Caps corresponding to an order of
> version numbers like described above (or to use a similar mechanism).
> 
>     *** Otherwise the technical solution *cannot* work! ***

But I assume we can still represent the "level of change" using
categories instead of version numbers?

> Installation of packages
> ========================
> In general there are
> - pre-install,
> - do-install,
> - post-install,
> - pre-remove,
> - do-remove,
> - post-remove,
> Scripts (they may be empty). They are triggered by performing an
> (package install/removal) Action.
> 
> Normally the install phase results into adding all provided Caps and the
> remove phase into the removal of them. Potential conflicts have to be
> solved *before* installation!

Personally I think we should strive to get away from scripts - they just
break. :)
Seriously.I would rather try to use a declarative approach as long as
possible and then only add hooks for scripts if we find out it is really
needed.

Scripts can also seriously trip us up and produce weird results that we
will have a hard time tracking down.
 
> Different kind of installs
> --------------------------
> There are different kind of installs:
> - upgrading,
> - full installation with a previous removal of an older version (if it
> exists).

Yes, currently SM actually differentiates these but only MC has the
capability of doing things "differently" when upgrading - and it figures
that out on its own today.

> Examples (for looking, if the concepts are working)
> ##################################################
> 
> Package WebBrowserStandard
> ==========================
> 
> - Provides Caps 'WebBrowser' and more;
> - needs Caps 'SocketStuff' and 'RenderingMachine';
> - exists in versions 1.1.3.1.7, 2.1.2.3.1, 2.2.1.1.1;
> - conflicts with package 'WebBrowserNicerButSlow' and some special
> SocketStuff versions.

I note that you have the Caps on the package and not on the releases.
This would mean they can't change over time, or rather - the releases
can't - at a specific point in time - have different Caps.

I also note you are expressing conflicts on package level, which also
suffers from the same problem.

> Each package version combo
>   WebBrowserStandard_1.1.3.1.7,
>   WebBrowserStandard_2.1.2.3.1, and
>   WebBrowserStandard_2.2.1.1.1
> needs a Transformation describing pre- and post Caps sets.
> 
> 
> E.g. for WebBrowser_2.2.1.1.1 there is the following:
> 
> a) It provides
> Caps 'WebBrowserStandard_2.2' being a super set including all Caps
> provided by WebBrowserStandard_2.1.2.3.1, but *not* all Caps provided
> by WebBrowser_1.1.3.1.7 (since the major number has changed);
> 
> b) It may require
> Caps
>   ('SocketStuff_3.4', 'RenderingMachine_2').
> A package RenderingMachine_2.1 would provide the last requirement.
> 
> c) It may conflict
>   - with package WebBrowserNicerButSlow in general (since it uses the
> same sockets); and
>   - with SocketStuff_3.4.3.7.x (arbitrary x), since the bugfix there had
> some very special side effect - unfortunately another installed
> important package PWorkaround needs exactly one of this versions of
> SocketStuff: it made a workaround not compatible with a newer version of
> it, where the side effect has been removed... (world is not perfect).
> 
> Note: if the user wants to install WebBrowserNicerButSlow or any other
> package, all packages have to be checked for conflicts.
> 
> How to model c)?
> - The conflict with package WebBrowserNicerButSlow is an all over
> package conflict and will be modeled as a conflict with Caps
> ('WebBrowserNicerButSlow'), which includes all Caps
> ('WebBrowserNicerButSlow_a.b.c.d.e') for arbitrary a, b, c, d, e.
> - The conflict with SocketStuff in versions 3.4.3.7.x is *not* an all
> over package conflict, it is a package-version conflict. This will be
> modeled as a conflict with Caps ('SocketStuff_3.4.3.7').

Ok, you differentiate with "all over package" conflicts and
"package-version conflicts".

> d) It has a Transformation
> requires:  ('SocketStuff_3.4', 'RenderingMachine_2.1')
> conflicts: ('SocketStuff_3.4.3.7')
>    |
>    V
> provides:  ('WebBrowserStandard',     "all over package Caps"
>             'WebBrowserStandard_2',   "major release"
>             'WebBrowserStandard_2.2', "minor release"
>             'WebBrowserStandard_2.2.1',
>             'WebBrowserStandard_2.2.1.1',
>             'WebBrowserStandard_2.2.1.1.1',
>             'WebBrowser'              "logical Caps").
> The last Caps in the result is a logical Caps provided by both packages
> WebBrowserStandard and WebBrowserNicerButSlow. Such a beast makes sense
> for expressing dependencies which shouldn't be bounded to a certain
> package: in this example there may be a point in the system
> automatically starting the installed WebBrowser, whatever it may be
> (WebBrowserStandard or WebBrowserNicerButSlow).
> 
> The question in case of holding package PWorkaround is, if there is a
> version SocketStuff_3.a.b.c.x with
>   a >= 4,
>   b, c, x arbitrary, but
>   not ((a = 4) & (b = 3) & (c = 7)),
> which also gives the input Caps 'SocketStuff_3.4'! Otherwise it wouldn't
> be possible to install WebBrowserStandard without removing PWorkaround
> first.
> So the package-version conflicts are reducing the number of packages
> capable of providing the needed input caps.

I am sorry Stephan but my brain didn't parse all this. I am fearing the
complexity here.
 
[SNIP of similar description of upgrades that also fried my brain :)]

[SNIP of make tool]

> To Lex's example
> ================
> 
[SNIP of description of example]
> Thinking in capability sets and a versioning policy is needed.

Well... I am not sure it is "needed", but that doesn't mean we can't
prepare for it. My summary is at the end.

> But testing is nevertheless important: it would be very good to have
> many automatically runnable tests to classify package combinations as
> stable/unstable/bleedingEdge/etc.. Since we are faced with a
> combinatorial explosion here, hints regarding
> stable/unstable/bleedingEdge/etc. from the package maintainers would be
> helpful though...

The stable/unstable categories are today mandatory per package release.
In other words - when you register a new release today you have to
specify maturity level AND compatibility level (giving something similar
to your version number scheme).


> Relationships with existing tools and concepts
> ##############################################
> 
> SM
> ==
> 
> If the proposed version policy would be used for classifying packages at
> SM, the release flag of packages would be replaced or modelled by a
> corresponding version number, which is much more fine granular.

You are saying "release flag". I assume you mean the "published" flag
(though I agree that it could have been called "release" or something).
And I don't agree. :) That flag is meant to tell people that even though
that release is available on SM it SHOULD NOT BE DEPENDED upon. The
maintainer might just be testing it etc. It is a "hey, this release
might not even be here - I may delete it at any second"-thing.

And also - the compatibility level categories recently introduced seems
very much like your version numbering scheme.

> An important question:
>   Who is responsible for maintaining logical packages?

You mean "Caps"?

> An - already known - example: packages WebBrowserStandard and
> WebBrowserNicerButSlow may both generate Caps 'WebBrowser' by an
> installation. There also may be a logical package WebBrowser which just
> depends on one of them. But who makes the choice? This is similar to the
> question, who decides which stuff is worth enough to come into the
> official distribution.
>  
> It would be interesting to enrich SM with a dependency mechanism as
> described in this paper...

Yes, at least in some regards - see my proposal/summary below.

[SNIPP of some stuff]

> Roadmap (loosely, may change)
> #############################
> 
> 0. Posting this mail and looking for the reactions.
> 
> *** If there is enough interest: ***

Yes, sorry for taking a while to respond. It overwhelmed me at first. :)
 
> 1. Creating a Swiki page with this stuff as a start.
> 
> 2. Getting a consensus for a versioning policy by
>   - getting feedback via ML discussions,
>   - posting an improved draft,
>   for some iterations.
> To come to a consensus at this point is *very* important,
> *independently* from the chosen technical solution!

Yes - or at least we need some form of concensus regarding the
structures in SM that can support different approaches.

> 3. Going further with discussions about technical ideas: the concepts
> should be good *before* coding. This doesn't mean to discuss forever and
> not to start with coding after a while...
> 
> 
> 
> Personal remarks
> ################
> 
> Don't try to be too perfect: KISS!

Yes. And I am sorry Stephan, but this model is... not really KISS in my
book. :) But I like parts of it.

> I have to admit that KISS seams to be quite complex in this domain...

Hehe. :)

> I'm willingly to invest some more time - e.g. by coding - into
> Dependencies for Squeak, but there should be a good chance, that
> the results won't be thrown into the trash can...

We should try to ensure that.

Ok, here is my summary (phew):

1. Caps is an extra "indirection layer" for expressing more abstract
dependencies. It should be possible to add "on top" of SM. It is a layer
I will not work with myself, because I think it is an overkill. But that
doesn't mean we can't make sure that the structures in SM allow for it
to be added.

2. Caps in themselves seems to be very easily modelled using
SMCategories. The category tree is currently centrally managed by me, in
other words - by the Guides. I assume that would be the same for Caps,
so it seems to fit.

3. Your version numbering scheme seems overly complex IMHO. I am not
sure if the current "Compatibility level"-as-categories-approach
fulfills your needs there. Note that the actual sub categories
representing each level can easily be changed - I just whipped up a few,
see
http://map1.squeakfoundation.org/sm/category/cb51b604-bb97-415d-a040-f23
1ecdd5bc7

Well, my head is spinning a bit, but what about this:

I have in my dev image added the concept of resources. A resource is
like an "attachment" to another SMObject. This way other tools can
attach metadata to for example SMPackages or SMPackageReleases that SM
doesn't really know how to handle - but doesn't care about. At this
point - since all modifications to the map must be done at the master
server - this would only be doable using a remote call to the master
server - but we can add those.

I then have a concrete subclass of SMResource called
SMPackageReleaseConfiguration. It holds a list of required
SMPackageReleases and a status flag. The flag can currently be #working
or #failing. #working means that the release we attach the config to
"works" with those specific required releases installed. It is thus a
tested configuration, as I have described before.

We also already have the mandatory categorizations of releases
"Compatibility level" and "Maturity level".

Now - let's say I continue with my "simpler" model that doesn't have
Caps and focuses on the SMPackageReleaseConfigurations as the basis for
figuring out "what do to". My mechanism is built in a separated
SMDependencyEngine. We may even figure out a set of "services" that such
an engine should be able to give.

One simply service would be "Ok, given that I want Seaside, Monticello
an Shout in this image - what specific installs and upgrades should I
perform and in which order?". This is btw the first service I am trying
to implement as a spike solution.

So if we agree on a set of such services - or in fact different engines
could offer different services and we can just let the engine hook
itself into the menues in the package tools and offer "what it has" -
then you can build your Caps-based engine on top of SM and use:

	- Caps as categories. They would then be optional to use by the package
maintainers.
	- Compatibility level (as exists today, mandatory)
	- Maturity level (as exists today, mandatory)
	- SMPackageReleaseConfigurations (as I have described and that I am
adding)
	- SMResource (so that you can optionally add extra metadata to
SMObjects inside the map)
	- And your engine can hook into the SqueakMap Package Loader menus when
installed.

Perhaps you are disappointed at me for not buying your solution - I hope
not. :) Caps is an interesting approach - even though I think it is
overkill. We share the idea of recording the compatibility level of
releases - but I don't like encoding it in version numbers. But I
desperately want SM to work as a "base" for testing different approaches
here, so please - tell me how this sounds and what you think/need to
consider moving forward with your idea.

regards, Göran