[squeak-dev] Squeak Package Management

Keith Hodges keith_hodges at yahoo.co.uk
Fri Jun 27 03:21:54 UTC 2008


Matthew wrote:
> Sake/Packages is an attempt to address both of the above
> concerns. It as yet has no UI. Unlike in SM or Universes, the
> entire distribution is versioned as a whole, rather than the
>   
This is a feature. If you want an old version of the s/p package map, 
then everything is as it was on that past date. (if certain criteria are 
adhered to)
> individual packages, so editing one package means making a new
> distribution version. This makes it easy to specify exactly what
> version of the database your image is up-to-date with, but does
> not scale if there are many contributions.
>   
Currently s/p uses mcz files to store its package definitions as a body 
of code.

I do not know of any squeak project that currently has scalability 
issues with too many contributors to the code repo. s/p is a body of 
code like any other.

If a scalability problem was to occur which I doubt very much indeed, 
there are numerous solutions.

1. split the packages up into smaller granularity, i.e. WebDevelopment 
packages, Dev Tools packages

2. adopt an editorial policy informally - i.e. load scripts only load 
the latest "approved" version.
(seaside users do this frequently, loading Lukas or Phillippe's versions 
in preference to unknown contributors.)

3. adopt an editorial policy formally - close the repo to a small number 
of approved editors.

4. spilt the package up, so that some packages load further package 
definitions as needed.

5. adopt a different server from the basic mcz server, such as Magma 
with more intelligent actions/merging capbability.

6. write a front end tool for publishing versions with some form of 
process management, aka, Gjallar / map.squeak.org / SqueakMap3

7. Split the package definitions master repo up into smaller repo's 
arbitrarily as needed. e.g. Database packages, Seaside, Kernel, etc.  
Use sake/tasks to auto generate combinations for specific squeak images 
based upon metadata. i.e. collect all definitions with the flag 
#worksInCroquet set into PackagesCroquet10

So yes it is scalable, because the actual nuts and bolts of the 
implementation of sake/packages is fairly orthogonal to how it is used. 
If the usage pattern turns out not to be scalable then we change the 
usage pattern.

i.e. Sake/Package definitons could be distributed via DeltaStreams 
rather than Monticello.

Randal wrote:
> Notice about 50 uploads a *day*.  And everything *works* in an
> automated way, except for the "CPAN gods" exceptions listed above
As I said before if the current solution cannot cope with 50 uploads a 
day, then it is not the fault of s/p, this is a server issue or a usage 
patterns issue. Squeaksource is not the only mcz server out there, and 
it is not the only solution for distributing/managing code. mcz can 
store diffs, magma might make a more intelligent back end and mc2 is out 
there somewhere too. This issue is orthogonal to s/p.

The s/p model is intended to be like that of wikipedia etc. relying upon 
open editorial policies. They cope with scalability issues somehow.

In addition those 50 uploads a day is quite high bandwidth. But I think 
that this would consist of people writing their packages, and every time 
they bump a single version, they have to republish so that their users 
can keep up.

Not so with s/p.

s/p allows publishers to publish 'NON-SPECIFIC' "the latest version that 
I have in my repo whatever it is". So package writers do not have to 
constantly worry about publishing their work for their bleeding edge 
users. This is a wonderful feature, you just publish ONCE even at the 
beginning of your project, you dont even need to wait until it is ready 
for release!

Users or publishers can write definitions that are more specific such as 
"the latest version that is authored by 'the master coder'. "

or even more specific such as "the latest version that is published on 
SqueakMap."

It's only when users find that a specific version is needed for a 
specific image that the definition needs to be updated with a "SPECIFIC" 
definition.  "Squeak3.10 definitely works with version 522 of Seaside2.8."

So...

A developer can publish a package, generating 100 versions as he goes, 
as it reaches some level of maturity the users add the definition that 
says version 100 definitely works in Squeak1.1.

Beta users who want the latest all the time, use

Packages beta load: 'ThePackageName'.

Normal users what the definitions that have been conformed to work BY 
OTHER USERS. They use..

Packages current load: 'ThePackageName'.

So... I doubt very much that the bandwidth for s/p will be anything like 
CPAN for the same amount of business.
> Sake/Packages - single point of update, making publishing hard
Single point of update, making management easy! And putting the power in 
the USERS hands, not the publishers.

The publisher will not have used the package in the specific images that 
the users have tested it in, so actually the Users collectively have 
more knowledge about the use of the package than the publisher. This 
model allows the user to publish his knowledge in the appropriate place. 
i.e A fix for Squeak 3.7 does not need to contaminate everyone elses 
definitions.

I would like to  know how CPAN publishes version 23 of the database 
driver for use on this version of Linux, with this Database loaded and 
this specialised file system. I beleive the database driver publishes 
his work and it is the USERS which decide how it will be used. S/P is 
primarily a means by which USERS share "What Works" information. 
SqueakMap and SqueakSource is where publishers can publish "What is 
Available".

Randal wrote:
> recent Squeak releases for installation, so we have a bootstrapping problem.
>   
I dont think s/p has a bootstrapping problem, since if your image is 
capable of loading code, s/p is only code and can be loaded any which 
way.  I surmse that s/p will have negligable bootstrapping problems when 
compared with the SqueakMap problems of the past. Furthermore 
LevelPlayingField is present to help with any bootstrapping problems we 
may have for any package in the future.

Randal wrote:
> And all of these systems are missing dependency and automatic upgrade
> management.
>   
s/p models dependencies to whatever depth you wish.

s/p has a rudimentary upgrade capability AND package UNLOAD capability...

If I want to unload Seaside it will unload in a sensible order every 
package that has been loaded which depended upon Seaside, prior to 
unloading Seaside (well that is the theory).

Damien Wrote:

> I don't see the point of having completely separate tools/websites for
> code management (SqueakSource/MC) and package management (all others).
> I want to configure an image as a user, then decide that I want to
> contribute on such package, and upgrade it to the devel snapshot with
> the same tool.
>
> Package management and code version control are really the same thing,
> just at different levels of granularity.
>   
Amen Brother!

If you fit the paradigm to the task it makes things much easier I think.

Let a coder use code to determine what and how his code is loaded into 
his image, and it will be a lot simpler than....

Using a web front end to a database of meta data, describing code files 
that are to be replicated all over the world for users that are still 
using 1980's modems. etc etc.

90% of software engineering is solving the right problem. The problem is 
actually very simple,

"collect the necessary data, what works where and with what dependencies."

s/p collects that data in an executable form (and all of the metadata is 
optional anyway), I dont see how it can get much simpler than that.

Randal wrote:
> * CPAN installation is easy
> * CPAN publishing is easy 
s/p publishing is probably easier than CPAN, since the package writer 
only has to publish once ever, and it is easy to browse every use case 
of a particular package...

load:  Installer install: 'PacakgesAllVersions'.

To find all versions of squeak which can load versions of seaside, do 
"implementers of #Seaside."
To find all packages that are dependant upon Seaside..

Packages current Seaside allDependants.

> Lowering this barrier in Squeak is important
>
> Merik

It may not be perfect but it is lowered to the point of actually being 
usable.

Ralph worte:
> A "feature" of Squeak that might differentiate it from perl is that
> there are many versions of Squeak.  Packages that work with one image
> might not work with another.  So, if you are using 3.8 then you do not
> necessarily want the latest version of some package if that package
> only works in 3.9 or 3.10.   PU addresses this problem by having a
> different Universe for each version of the image, though I think there
> is currently a Universe only for 3.9 and 3.10.  Even if the code for
> PU could run in earlier images, it isn't useful until someone builds a
> Universe for it.
Got it exactly. CPAN does not address the real problems that we actually 
face. i.e "WHAT works WHERE." as seen by package users.  And this is 
exactly why it is that Users of a particular image that need to manage 
the package definitions for that image. The package writers are often 
behind the enlightened users who are putting their package to new and 
varied uses in different contexts.

> Maybe PU is the closest to CPAN we have.  So the question is, do we keep
> mutating PU into what I'm looking for? 
No, because PU will be no use in an image that doesnt have any Network 
code, doesnt have XML parser, doesnt have Morphic, so if you do improve 
PU someone will have to develop another solution for the more stripped 
down image. If future squeaks are based upon more stripped down images 
then we end up with two solutions to the same problem.
>  Or do we first need to agree that the
> CPAN might be a pretty darn good model?
>   
Its not a good model because it is solving a problem that we don't 
currently have. We are not distributing tar files for users around the 
world that are using modems to access the internet at 1200 baud, for use 
in a language with no reflexive capabilities. i.e. unix bash/csh etc

(btw: Our packages do not have to be restricted to code, they could 
include objects, projects, or anything loadable or creatable by code, 
this includes mantis fixes.)

Our problem is not with publishing packages, our problem is enabling 
users to load what THEY want into their images and to share their 
feedback as to what works with other users in a way that can be used.

Ralph wrote:
> CPAN is probably a good model.  However, a model that works well for
> one language does not always work well for another.  How would you
> think a CPAN-like system would handle the fact that people are using
> 3.6, 3.7, 3.8, 3.9 and 3.10 and that some packages work for all of
> these, but some packages are specialized for a particular image?
>   
Exactly, that's why CPAN is not a darn good model for this problem. I 
can only envisage that it would be a darn big complicated unmanageable 
pain. The proposal is made with blinkers on, since Randal refused to 
even look at s/p because of his concerns over "scalability", which as I 
have described above is a complete red-herring-issue.

Randal wrote:
> We do need something that says package X works with squeak Y and package X
> depends on package Z.  so it's two dimensions, not just one.
>   
Yes we do... and which omnipotent package publisher knows the answer to 
that question in every context? Answer none, problem is solvable with a 
user feedback mechanism, not a package publishing mechanism.

Merik wrote:
> And what about patches?
>
> When an Seaside website needs a patch of an Integer method, for example.
Yes indeed , some images need patching before a particular package will 
load. Sake/Packages handles this, it can load patches from mantis using 
Installer ensureFix: 1234.  OR It can specify a dependency upon the 
Sake/Tasks load mantis fix task, which documents the fixes loaded as it 
goes.

so Randal why you wont look at Sake/Packages is beyond me.

regards

Keith


























More information about the Squeak-dev mailing list