Hi Goran and Lex,
Hi other Squeakers,

there are following some comments about how my DEPS model could work
into the light of some of the thoughts from the BIG posting.

Note: in the following DEPS means the corresponding paper contained in
the mail subjected
  [DEPS][PAPER] Dependencies for Squeak
sent to the list recently.

goran.krampe at bluefish.se wrote:
> Hi Lex!
> ...

> lex at cc.gatech.edu wrote:
> 
>>Hey Goran, ...

>...

> Btw, you did stumble upon a slightly interesting fact though - you said
> "how do you fit post-release bug reports into the world view?". Given
> that in the planned SM the dependency information is not embedded in the
> actual release (as in Debian) this means that the dependency information
> can be revised post-release. And it should, especially by adding more
> working know configurations. If the maintainer or anyone else discovers
> that, darn - there was actually a serious bug in Z1.2 and Z1.3 is really
> needed - then the configuration can and SHOULD be changed.

This could be modeled as updated Transformations in DEPS.

> 
> And if someone discovers that, hey "I used Z2.0 and it seems to work
> fine with that too!" then that should/could be added too. That is one of
> the key points of NOT embedding dependency information inside the
> releases.

This could also be modeled as changed Transformations: see section
"Technical terms -> Caps -> Boolean logic".

> 
> 
>>Aside from the issues with the "guarantees" idea, it has not been
>>established that the jigsaw puzzle of version-specific dependencies is
>>going to be practical for tools to work with.  This is a significant
>>technical problem that versioned dependencies add, and everyone seem to
>>be waving it off as something to handle in the future.  I don't know
>>that this is solvable, guys, and no one has made a convincing argument
>>otherwise.

I see it more as a *social* problem, to have responsible maintainers
trying to do their best. But if a maintainer wants his/her package to be
in widespread use - which would give some happyness, at least for me -,
then I think he/she is doing his/her best
- to make it compatible with other packages other people are using,
- to be careful with a potential 'stable' categorization,
- to define its dependencies carefully.

And afterwards it is no problem to add conflicts...

Another idea: there also could be some rating system for packages,
marking white and black sheeps!

>...

>>dependency jigsaw cannot be adequately solved.  Let me run you through a
>>small example.  Packages A and B depend on Collections:
>>	
>>	A 1.0   needs   Collections 1.0
>>	B 1.0   needs   Collections 1.0
>>
>>Fine so far, I install all three packages.  Now the collections library
>>gets updated to 1.1.  I cannot install the upgrade without breaking the
>>dependencies of A and B!!  So, I wait before installing it, even though

I have treaten this example (further parts are snipped here) in DEPS.

> 
> 
> Not entirely true. First of all - the engine should be parameterizable
> (cool word) by you.
> 
> So the idea is that if you - at this point - select Collections 1.1 and
> press "install" it could say something like "The installed package
> releases A1.0 and B1.0 are only known to work with Collections1.0, but
> since Collections1.1 is marked by the maintainer as being 'Code changes,
> but only big fixes' they could still likely work, do you still want to
> proceed and install it?"

The proposed versioning number system in DEPS is just a model to express
such marks like 'Code changes, but only big fixes' technically. Though I
more have had *small* fixes in mind as an example for changing the 4.
version number...

> 
> And if you set a parameter, let me fantasize here - "Allowed
> compatibility threshold" or something - then perhaps it wouldn't even
> ask if the new release was below a certain compatibility level.

Again I think it is important to mention the social aspects here: as
better the package maintainers categorize their packages, as better such
a technical solution could work, if there is only this thing.
But there may be more to help the technical solution computing good
results, e.g. prefiltering of packages after some criteria:
- stable/unstable,
- rating by users,
- rating by automatically testing,
- trust measure of package maintianer (somewhat delicate),
- etc..

> 
> Now, it doesn't end there. Not only can you install Collections1.1 (and
> as I described in a much more informed way than if the recorded
> dependencies were just "A and B needs Collections") but if someone ELSE
> has already tried this, and discovered A annd B works just FINE (who
> knows, perhaps there is even a test suite to run!) then he/she can have
> beaten the maintainers of A and B respectively and attached new
> configurations to A and B that says that, according to him - A1.0 works
> just fine with Collections1.1 (and the same for B).

In DEPS to be modeled by Boolean logic in requires.

> 
> So if you then trust this person, you can go ahead with even more
> information to guide you - he said it worked, and since it was Dan
> Ingalls - you decide to trust him.

This is an interesting point: rating 'configurations' after the people
making them.
Note to 'configurations': in DEPS these would be expressed as
Transformations expressing the dependencies, possibly enriched by a
pointer to the respected person together with its *installed* packages
to get the whole thing. In other words: who has used which
Transformations to get its fancy system configuration.

> 
> And my third final point - if there was no Ingalls-config available, and
> you did have to "take a chance" - then YOU can be the good citizen
> paving the way for others to follow by verifying it works and attach a
> new configuration to A and B.

But its expressiveness is limited, if the knowledge about *all* the
installed packages hasn't been recorded, too. YOU can have a
configuration with very few packages totally incompatible with many
others in widespread use, for example.

> 
> Now - IMHO this is a GREAT model :). I am not saying we will not
> discover details to tweak with it - for example, I can surely see it
> extended with information about "intended dependencies" - meaning that
> the maintainer might want to tell people what packages he wants to
> depend on, this may not always be the same thing.

In DEPS the "intended dependencies" would partly modeled by the
compatibility level expressed by the version number given with the
package. The other part would be Transformations for installing it,
first given by the maintainer in addition to the package (there may be
some process, too...). Later it should be possible to change the
Transformations: normally they would be enriched with conflicts. I
currently don't think it makes much sense to change a given version
number without changing the package, too, but there may be exceptions (I
think this will rarely be used in Debian for overriding the normal
dependency mechanism).

> 
> 
>>I might be the main developer of package C, which also uses Collections.
>> Okay, so eventually tthe author of A, being a great citizen, upgrades
>>their Collection package and reruns their tests -- shock, nothing brock.
>> While they are at it, they add a few class comments, and then post A
>>1.1 which depends on Collections 1.1.  Drats, I still cannot update my
>>Collections library, because that will make me uninstall B.
> 
> 
> No big difference, at this point - select Collections 1.1 and press
> "install" it could say something (quite similar) like "The installed
> package release B1.0 is only known to work with Collections1.0. A1.0 on
> the other hand can be upgraded to A1.1 which is known to work with
> Collections.1.1. Since Collections1.1 is marked by the maintainer as
> being 'Code changes, but only bug fixes' B1.0 could still likely work,
> do you still want to proceed and install Collections1.1 and also A1.1?"
> 

> Of course, exactly how the engine will talk to the user is a UI issue.
> :) And upgrading A1.1 would of course be optional - the engine should
> only be HELPFUL - not forceful. You should always be TOTALLY FREE to
> install what the heck you like.

This is an interesting point: how to *override* the automatically chosen
solution?
In DEPS a conflict is a conflict: it makes no sense to override a known
conflict. But what about a weak conflict? E.g. Foo_3.2.1.1.1 *may* break
Bar under some not very likely circumstances.

Straightforward idea (after some moments of thinking): classifications
of conflicts and using them together with the user install policy as a
filter together with hints to the user.

More simple solution: not stating the conflict in the
DEPS-Transformation, but writing a warning (possibly standardized, so
that GUIs could issue it automatically) into the description of the
package. Then the user has the choice...

>...

> As I have clearly shown this is not the case. I have NEVER argued for
> the dependency engine to be some kind of POLICE. You can still install
> *whatever you like*. The whole point is that you can do this and be
> AWARE of what it means and what the risks are.

Additional idea: policy ignoring conflicts in DEPS-Transformations as
option to force an install.

> 
> And I have also, I hope, clearly shown that since we can all attach
> tested working configurations you will not be stuck at the whim of the
> maintainers.

> And I also hope i have shown that the configurations can be
> added AFTER the release,

I think of introducing some whatever dependency system slowly: at first
only a few packages get their technically usable version numbers. Then
the first package is published with separately delivered
DEPS-Transformations describing its dependencies to these few packages,
so they can be automatically installed with it. This is so comfortable
that more and more package maintainers jump onto the same train...

> and even MODIFIED after the release.

This is a crucial point: a package should contain its expected
compatibility level (via its version number), but not its dependencies.

> This is a
> GOOD THING - because if they are wrong they should simply be fixed.

Agreed.

> ...

> Of course, it all gets "worse" in some kind of way
> with lots of intertwined packages - but I don't think the model will
> suffer. There are some really nice things here helping it:
> 
> 1. Users can help keep the dependencies tracked and verified, not only
> maintainers. This is cool.

Here again I'm seeing some social questions: who is responsible to
change the 'official' dependencies which the non expert (or an expert
not knowing the 1000th package) user of Squeak *normally* uses? Since
he/she wants to have the one-click install...

> 2. Pressure can build on key packages that really needs to be upgraded
> in order to work properly.
> 
> This leads me to something I haven't talked about before - it should be
> possible to record an "anti configuration" - meaning that "No, sorry, I
> tested A1.1 with Collections1.2 and nope, didn't work". This would be
> invaluable to have. Again, we record information - that is all. Then how
> the information is used is purely up to you. I can even imagine
> different engines or at least clearly different pluggable strategies. :)

Recording the "anti configuration" makes sense.

But I think while writing DEPS I've had a more rigid policy in mind.
Assumed the observation above is true: then it is a *conflict*, isn't
it? Which *normal* user not trying to fix the bugs (say > 99% of all
users) would ever like to install an "anti configuration"?

So this "anti configuration" should be expressed as a conflict to
protect the normal user. For the few experts, which are not only
experts, but also interested in this special broken "anti configuration"
the override buttons should be there (but switched off in the official
image).

> 
> 
>>acceptible that every package needs to be tested with each incremental
>>version of every other package, and thus achieve the lockstep
>>progression?  That seems very inefficient if most package updates are
>>compatibility-preserving bug fixes.
> 
> 
> I have described that the compatibility level categorization of releases
> will make this information explicit.

> If you don't want to test your
> releases at all - then fine, others will do it for you. :)

The maintainer has made at least a minimal test, if there has been just
one run of the package...

To an automatically dependeny resolving system: I think such a bleeding
edge package should just been outside automatically installation, since
it would make more problems than it helps. A few of such packages and
you can start with a fresh image...
The good thing: no responsible maintainer of another package would put
such a bleeding edge package in the requires of his/her package! So
possibly this is already the automatical solution of this policy problem
given without any costs...

> And if you do
> test them, then why not record which releases you tested it against?

This certainly makes sense.

>...

> And it will also be very suitable for test driven development. Did you
> hear that Stephane? :) Because if we can attach or in some other way
> associate how to run the tests with the packages (included or not) then
> they will form a formal basis of testing out new configurations of
> releases.
> 
> Hell, it could even be automated in theory - we could have a robot
> server sitting trying to test new combinations of package releases and
> attaching those autotested configurations automatically.

This would be very nice!

To the questions of single or multiple tree or not tree'd maps
--------------------------------------------------------------
A dependency system should work with distributed maps as well.
There should be distributed dependencies then, too; in the tree case: a
local server surely has a map containing some local packages (why to
have it otherwise?), and therefrom it has at least some dependencies
between packages to be expressed, which its master doesn't know, if a
dependency system should be used for these local packages.

>... <snipped very much here>


Greetings
Stephan
-- 
Stephan Rudlof (sr at evolgo.de)
   "Genius doesn't work on an assembly line basis.
    You can't simply say, 'Today I will be brilliant.'"
    -- Kirk, "The Ultimate Computer", stardate 4731.3