Where we are going with partitioning and harvesting etc.

[goran at krampe.se]

Hi guys!

"Doug Way" <dway at mailcan.com> wrote:
> On Tue, 21 Jun 2005 15:00:01 +0200 , goran at krampe.se said:
> > I had my sights set on this (brief description):
> > 	- Get Universes and SM playing together. As promised by me to Lex and
> > the team.
> 
> Still true.  The iSqueak stuff is *only* relevant to the Basic packages
> making up the base image, not the Full packages or external packages. 
> And we could still have Universes for the Basic packages.

Sure, I agree basically, but see below.

> > 	- Use PIs to partition the image. The good ol TFNR idea, which Ken now
> > also expresses in his post.
> 
> Still very true, this is exactly what the iSqueak partitioning does,
> except the PI packages also happen to be MC packages, which have some
> big benefits IMO in terms of tool support.

Yes, I am aware of that - and agree, it is the same to the extent that
PI is used.

> > 	- Tie the PIs to corresponding SM entries. This is key to get the full
> > connection to maintainers and their emails etc.
> 
> I think this is still true, and worthwhile.  SM should still be the
> place to get maintainer names, email addresses and other miscellaneous
> meta-info for all packages, including Basic packages.  Now, how exactly
> they are "tied" together I'm not sure... perhaps by naming convention
> (having the same name), or by virtue of the SM entry pointing to the
> same .mcz file, or something else.

There has been for a while a text field in SMPackage that simply names
the PI.
And this is even used today by the PI-changesets-extras-package that Ned
has put together that among other things include some code from me.

> > 	- Gradually move to maintaining these "image packages" as MCs "released
> > on SM", instead of using changesets in the stream. This part obviously
> > is different from the recent plans in the Package team.
> 
> Yes, this is the one part that is different.  In the proposal, the idea
> is to simply have a special MC repository for the 3.9 release as it goes
> through alpha-beta-final, separate from the various master repositories
> for each package. (this repository is a sort of "configuration", similar
> to the Universe concept, actually)  The alpha update stream gets the
> latest package versions directly from this MC repository rather than
> through SqueakMap.
> 
> The thing I really like about this is that the MC tools are there in the
> iSqueak image, showing you which package versions are in the image, you
> can browse the code for a package, and you can commit a new package
> version right there in the tool if you wanted to... no need to go to the
> SM website and upload your new version.  For this sort of alpha
> stream/Basic package development with potentially lots of integration
> fixes, it seems like SM would add an extra layer which isn't strictly
> necessary.  In any case, it works as-is right now, Impara is using it.

I agree that going straight to MC like you describe is perfect during
development but I would have hoped that all packages in the released
image do have "releases" on SM and do have SM counterparts. In other
words - the above would be fine with me as long as we make proper SM
releases at some points in time, like say we could do as Debian does -
have a freeze period where we freeze package by package (=make SM
releases) on our way towards the release of Squeak.

A situation where we have two different kinds of packages seems quite
bad to me.

Use case: I want to see what versions of packages I have in my image, I
go to the SqueakMap Package Loader and select "installed". Now, if there
are *other* packages that don't show up then seems quite silly.

I can come up with tons of use cases where this division is A-packages
and B-packages is bad.

And note that all I want is:

	- That each PI package in the image has a corresponding SM package.
	- That we "tie" it using the PI name field (trivial).
	- That we do make SM releases at least at some points in time.

Doing intermediate MC releases during the whole development phase is
perfectly fine. And I would also like to add a "Repository" field to
SMPackage that points to the repository where other non-released
versions can be found.

> I was thinking that the final 3.9 Basic packages would all be on SM, of
> course, but that the interim 3.9alpha packages would not need to be.
> (unless a package owner felt like putting them out there)

See above. I think all should "be" on SM, but that doesn't stop anything
AFAICT in this plan.

> Actually, maybe there is a way to work SqueakMap into this process, to
> unify things a bit more.  Then perhaps the latest 3.9alpha
> "configuration" could be an SM universe instead, without requiring a
> special MC repository.  As long as this was done in a way which did not
> affect the MC tools or make the development/integration process any more
> difficult, that might be good.  That might be something to work on after
> we get this going.

Yes, something like that. But as you say, nothing we need to decide now
UNLESS the plan aims to use MC "configurations" or whatever - because
then we are sidestepping SM in this area.

> But in any case, I think we're ready to try this proposal as-is right
> now, as there is no extra coding work to be done, it should work as soon
> as we set up a repository.
> 
> > 	- Add my planned dependency model in parallell with Universes. They in
> > fact complement each other quite nicely when thinking more closely about
> > it.
> 
> Yep, that still sounds good.  AFAIK, there is not a dependency model
> within this iSqueak plan, it's simply an MC repository acting as a
> config map.

Ok, I thought there was some kind of "configuration" thing going.

> > Now it seems like the plan is to move to using something independent of
> > SM, which means I have no idea on how to proceed with the above. Perhaps
> > I am not interpreting the plan correct - then you all may of course
> > explain it to me and slap me. :)
> 
> Hopefully my replies above cover this? :)  It is sort of independent of
> SM for now, but it's just the Basic image packages.

Ok, I would still very much for us to do as I described above - making
sure there are SM packages for each of these packages (with PI filled
in). That can't be a bad thing.

> > I have also some serious doubts about schemes/models that depend on
> > "community work". With that I mean work "by" the community "for" the
> > community, eh, if you know what I mean - instead of work "by" yourself
> > "for" yourself. Hmmm, Ken has a much better english than me, but he is
> > essentially saying the same thing.
> 
> Yes.  This proposal is a big step away from the monolithic image and
> should rely less on "community work" than the current setup.  In that
> respect I think it's not much different than the partitioning you had in
> mind.  Most of the active package development should be done by the
> package owners in the master repositories for the packages, not in the
> 3.9 release repository.

> The 3.9 people would just assemble the latest
> packages which work together, and maybe occasionally work on some
> integration fixes (or get other people to do this).
<
> Maintaining a release repository (or a Universe) is some work, but a lot
> less than maintaining everything as we do now. ;)  Ok, it's a bit
> different than Universes, but you get the idea.

Yes, but I still get nervous because it still sounds like a centralized
plan somehow implying "a group of people doing work for all"-model which
I am very scared of at the moment.

> > So my view is that we need a system which relies on such individual work
> > only, or at least to 95%.
> 
> Yep, I hope that would be true for this proposal.

Ok. Good. Then we will just see where it goes.

> > Ken asked "What are we waiting for?" and... I am not sure. The
> > PI-partitioning bit - which is key here - should simply "be done" IMHO.
> > I say let's get *a single person* to just do it and push a .cs into the
> > stream which creates the PIs. I was inclined to get Doug to do it (being
> > v3.9 boss and all), but I don't really care as long as someone does it.
> > Again, I am not sure how this plays with the latest plans/developments
> > (read the packages archive for details).
> 
> It still fits in... I can add the partitioning changeset (from iSqueak)
> to the restarted 3.9alpha stream as soon as we have a SqueakSource/MC
> repository set up for 3.9.

Ok, can you get in contact with me as soon as this is done so that I can
help us get SM packages in place?

> - Doug

regards, Göran