Where we are going with partitioning and harvesting etc.

[Doug Way]

On Wed, 22 Jun 2005 12:23:29 +0200 , goran at krampe.se said:
> (snip)
> > > 	- Tie the PIs to corresponding SM entries. This is key to get the full
> > > connection to maintainers and their emails etc.
> > 
> > I think this is still true, and worthwhile.  SM should still be the
> > place to get maintainer names, email addresses and other miscellaneous
> > meta-info for all packages, including Basic packages.  Now, how exactly
> > they are "tied" together I'm not sure... perhaps by naming convention
> > (having the same name), or by virtue of the SM entry pointing to the
> > same .mcz file, or something else.
> 
> There has been for a while a text field in SMPackage that simply names
> the PI.
> And this is even used today by the PI-changesets-extras-package that Ned
> has put together that among other things include some code from me.

Ah, I forgot about that.  It makes sense to use that, then.

> > > 	- Gradually move to maintaining these "image packages" as MCs "released
> > > on SM", instead of using changesets in the stream. This part obviously
> > > is different from the recent plans in the Package team.
> > 
> > Yes, this is the one part that is different.  In the proposal, the idea
> > is to simply have a special MC repository for the 3.9 release as it goes
> > through alpha-beta-final, separate from the various master repositories
> > for each package. (this repository is a sort of "configuration", similar
> > to the Universe concept, actually)  The alpha update stream gets the
> > latest package versions directly from this MC repository rather than
> > through SqueakMap.
> > 
> > The thing I really like about this is that the MC tools are there in the
> > iSqueak image, showing you which package versions are in the image, you
> > can browse the code for a package, and you can commit a new package
> > version right there in the tool if you wanted to... no need to go to the
> > SM website and upload your new version.  For this sort of alpha
> > stream/Basic package development with potentially lots of integration
> > fixes, it seems like SM would add an extra layer which isn't strictly
> > necessary.  In any case, it works as-is right now, Impara is using it.
> 
> I agree that going straight to MC like you describe is perfect during
> development but I would have hoped that all packages in the released
> image do have "releases" on SM and do have SM counterparts. In other
> words - the above would be fine with me as long as we make proper SM
> releases at some points in time, like say we could do as Debian does -
> have a freeze period where we freeze package by package (=make SM
> releases) on our way towards the release of Squeak.

Yes.  I did mention that at a minimum we would have the 3.9 final
packages released (or "published") on SM, and I forgot to mention that
of course we should have the newly partitioned 3.8 packages on SM as
well.  And then, since there may be several months between 3.8 and 3.9,
it might make sense to periodically release the current set of alpha
packages to SM, if they are determined to be in a stable state.  How
often exactly, I'm not sure.

Also, keep in mind that this is just putting package sets on SM from the
3.9 "release repository".  Each package should still have its own master
repository, where core package development happens.  The package owner
might also be putting package versions from there on SM as well. 
(Hopefully there will not be issues with versioning numbering in MC with
the two types of repositories.  I haven't really used MC working across
multiple repositories to know how it's handled.  If it matters, I guess
the based-on-iSqueak 3.9alpha image would list two repositories for
every package in the MC Browser, the master repository and the release
repository.)

> A situation where we have two different kinds of packages seems quite
> bad to me.
> 
> Use case: I want to see what versions of packages I have in my image, I
> go to the SqueakMap Package Loader and select "installed". Now, if there
> are *other* packages that don't show up then seems quite silly.

Good point.  We would not be adding brand-new named packages to the
release repository (3.9alpha update stream) without also adding them to
SM.  (By this I mean a entirely new package, not a package version.) 
So, the only issue here would be that the SM Package Loader could
possibly show an incorrect older version of a package that was updated
via MC.  But if you have the SM package point to the PI, perhaps the SM
Loader could try to be "smart" about what is really in the image? 
Although, if the particular package version was a development version
only in the MC release repository and not in SM, the SM Loader couldn't
show an SM version, of course, perhaps it would just show the MC
version.  (The SM version numbers may end up mirroring the MC version
numbers anyway.)

> I can come up with tons of use cases where this division is A-packages
> and B-packages is bad.
> 
> And note that all I want is:
> 
> 	- That each PI package in the image has a corresponding SM package.
> 	- That we "tie" it using the PI name field (trivial).
> 	- That we do make SM releases at least at some points in time.

Agreed 100%.  If we tie them together using the PI name field in the SM
packages, that should work.  (I don't think we want the PI itself to
point to the SM package.)

So we agree on this.

As a side note, I'm not sure if this means that we may not need Ned's PI
additions (for filing out PI's), if we're always using MC... I'm trying
to remember exactly what the additions were.  Although if they are
generally useful and harmless, we could just add them.

> Doing intermediate MC releases during the whole development phase is
> perfectly fine. And I would also like to add a "Repository" field to
> SMPackage that points to the repository where other non-released
> versions can be found.

Sure.  I guess this would point to the "master repository" for the
package, not the release repository.

> ...
> > Actually, maybe there is a way to work SqueakMap into this process, to
> > unify things a bit more.  Then perhaps the latest 3.9alpha
> > "configuration" could be an SM universe instead, without requiring a
> > special MC repository.  As long as this was done in a way which did not
> > affect the MC tools or make the development/integration process any more
> > difficult, that might be good.  That might be something to work on after
> > we get this going.
> 
> Yes, something like that. But as you say, nothing we need to decide now
> UNLESS the plan aims to use MC "configurations" or whatever - because
> then we are sidestepping SM in this area.

Er, ah, um.  Well, the update stream example Andreas gave did use
MCConfiguration, so perhaps there is some duplication of effort here. 
Although that is only needed when a "config map" changeset needs to be
added to the update stream to bring us up to a certain set of packages,
in order to execute a DoIt changeset (to massage instances in the image
for example), or if something must be loaded in a certain order.

But still, these configurations would only be within the
alpha-development MC release repository, I believe.  I'm not sure they
could be extended to the world of packages out there, and in that sense
I'm not sure they could compete with SM configurations.

Perhaps there is a way we eventually could replace the MCConfiguration
command in the example update with an equivalent SqueakMap command, if
all development package versions were on SqueakMap.  But I'm not sure if
it's worth the effort.

Stephane emailed me earlier saying that he was in favor of this MC
partitioning plan, and that he thought of MC as more appropriate for
package development, and SM for package deployment, which seems roughly
right to me.  And I think we're really talking about development here. 
But yeah, the concept of configurations/dependencies can apply to both,
so there is some potential overlap.  (I don't mind mentioning Stephane's
private email on the list here as he is known to do the same thing. ;-)
)

> ...
> > Hopefully my replies above cover this? :)  It is sort of independent of
> > SM for now, but it's just the Basic image packages.
> 
> Ok, I would still very much for us to do as I described above - making
> sure there are SM packages for each of these packages (with PI filled
> in). That can't be a bad thing.

Agreed.

> ...
> > The 3.9 people would just assemble the latest
> > packages which work together, and maybe occasionally work on some
> > integration fixes (or get other people to do this).
> <
> > Maintaining a release repository (or a Universe) is some work, but a lot
> > less than maintaining everything as we do now. ;)  Ok, it's a bit
> > different than Universes, but you get the idea.
> 
> Yes, but I still get nervous because it still sounds like a centralized
> plan somehow implying "a group of people doing work for all"-model which
> I am very scared of at the moment.

True.  Only the cross-package changes (integration fixes and such)
should have to happen in the centralized release repository.  Hopefully
most of the work will be decentralized in the individual packages by the
maintainers working in their own repositories.  (Actually, I can imagine
detangling-type work would need to be done in a release repository and
then fed back to the package owners.)

One cool thing about this plan, I think, is that if the centralized part
of it completely falls apart for some reason, you'll still have all of
the partitioned packages out there in individual repositories, with some
versions on SqueakMap which you could still assemble via Universes or
whatever, which wouldn't be too far from your original plan.  (It might
just take a really long time to put together a working 3.9 then. ;) )

> > > So my view is that we need a system which relies on such individual work
> > > only, or at least to 95%.
> > 
> > Yep, I hope that would be true for this proposal.
> 
> Ok. Good. Then we will just see where it goes.

Yep. :)

> > ... I can add the partitioning changeset (from iSqueak)
> > to the restarted 3.9alpha stream as soon as we have a SqueakSource/MC
> > repository set up for 3.9.
> 
> Ok, can you get in contact with me as soon as this is done so that I can
> help us get SM packages in place?

Yes, will do.  It looks like Cees & Avi have the SqueakSource/MC
repository nearly set up, then we can add the partitioning changeset to
the restarted 3.9alpha stream.

- Doug