Where we are going with partitioning and harvesting etc.

[Doug Way]

On Tue, 21 Jun 2005 15:00:01 +0200 , goran at krampe.se said:
> Hi all!
> 
> (cross posting since Ken's recent post touches on the very same aspects)

Hi Goran & all, I'll cross-post for this one also since it affects all
three groups.  (I missed Ken's earlier post as I hadn't checked the
Janitors list in a couple of days.)

> I have been re-reading the archive (=the packages team archive) to see
> where we are and what I had promised to do. :) I am a bit unsure about
> the latest development.

If I understand everything correctly with the latest
partitioning/iSqueak developments, I think your to-do items haven't
changed too drastically... more below.

> I had my sights set on this (brief description):
> 	- Get Universes and SM playing together. As promised by me to Lex and
> the team.

Still true.  The iSqueak stuff is *only* relevant to the Basic packages
making up the base image, not the Full packages or external packages. 
And we could still have Universes for the Basic packages.

> 	- Use PIs to partition the image. The good ol TFNR idea, which Ken now
> also expresses in his post.

Still very true, this is exactly what the iSqueak partitioning does,
except the PI packages also happen to be MC packages, which have some
big benefits IMO in terms of tool support.

> 	- Tie the PIs to corresponding SM entries. This is key to get the full
> connection to maintainers and their emails etc.

I think this is still true, and worthwhile.  SM should still be the
place to get maintainer names, email addresses and other miscellaneous
meta-info for all packages, including Basic packages.  Now, how exactly
they are "tied" together I'm not sure... perhaps by naming convention
(having the same name), or by virtue of the SM entry pointing to the
same .mcz file, or something else.

> 	- Gradually move to maintaining these "image packages" as MCs "released
> on SM", instead of using changesets in the stream. This part obviously
> is different from the recent plans in the Package team.

Yes, this is the one part that is different.  In the proposal, the idea
is to simply have a special MC repository for the 3.9 release as it goes
through alpha-beta-final, separate from the various master repositories
for each package. (this repository is a sort of "configuration", similar
to the Universe concept, actually)  The alpha update stream gets the
latest package versions directly from this MC repository rather than
through SqueakMap.

The thing I really like about this is that the MC tools are there in the
iSqueak image, showing you which package versions are in the image, you
can browse the code for a package, and you can commit a new package
version right there in the tool if you wanted to... no need to go to the
SM website and upload your new version.  For this sort of alpha
stream/Basic package development with potentially lots of integration
fixes, it seems like SM would add an extra layer which isn't strictly
necessary.  In any case, it works as-is right now, Impara is using it.

I was thinking that the final 3.9 Basic packages would all be on SM, of
course, but that the interim 3.9alpha packages would not need to be.
(unless a package owner felt like putting them out there)

Actually, maybe there is a way to work SqueakMap into this process, to
unify things a bit more.  Then perhaps the latest 3.9alpha
"configuration" could be an SM universe instead, without requiring a
special MC repository.  As long as this was done in a way which did not
affect the MC tools or make the development/integration process any more
difficult, that might be good.  That might be something to work on after
we get this going.

But in any case, I think we're ready to try this proposal as-is right
now, as there is no extra coding work to be done, it should work as soon
as we set up a repository.

> 	- Add my planned dependency model in parallell with Universes. They in
> fact complement each other quite nicely when thinking more closely about
> it.

Yep, that still sounds good.  AFAIK, there is not a dependency model
within this iSqueak plan, it's simply an MC repository acting as a
config map.

> Now it seems like the plan is to move to using something independent of
> SM, which means I have no idea on how to proceed with the above. Perhaps
> I am not interpreting the plan correct - then you all may of course
> explain it to me and slap me. :)

Hopefully my replies above cover this? :)  It is sort of independent of
SM for now, but it's just the Basic image packages.

> I have also some serious doubts about schemes/models that depend on
> "community work". With that I mean work "by" the community "for" the
> community, eh, if you know what I mean - instead of work "by" yourself
> "for" yourself. Hmmm, Ken has a much better english than me, but he is
> essentially saying the same thing.

Yes.  This proposal is a big step away from the monolithic image and
should rely less on "community work" than the current setup.  In that
respect I think it's not much different than the partitioning you had in
mind.  Most of the active package development should be done by the
package owners in the master repositories for the packages, not in the
3.9 release repository.  The 3.9 people would just assemble the latest
packages which work together, and maybe occasionally work on some
integration fixes (or get other people to do this).

Maintaining a release repository (or a Universe) is some work, but a lot
less than maintaining everything as we do now. ;)  Ok, it's a bit
different than Universes, but you get the idea.

> Harvesting is a typical example, an unselfish act for the good of all -
> not directly connected to any personal immediate need. Or trying to
> maintain a Universe, yes, I *do* have doubts about that model - but at
> the same time I want to enable it - because I may very well be wrong :).
> 
> Or staking out the image, an act which has stalled numerous times for
> that specific reason I think. All these activities have that in common -
> they are "non individual" activities relying on community members to do
> the work "for the benefit of all".
> 
> I have come to the personal conclusion that schemes depending on such
> work have a high risk of failure in the Squeak community. We simply
> aren't that good at it. We are much more individualistic and we like to
> scratch our own itches, which means there are two things we are really
> good at:
> 
> 	- Maintaining our *own* packages, typically driven by our own needs in
> combination with pride and "feeling good" when other people appreciate
> your work.
> 
> 	- Finding and fixing bugs in other Squeakers' packages but almost
> always driven by our *own* projects/packages, we simply fix stuff we
> stumble over.
> 
> So my view is that we need a system which relies on such individual work
> only, or at least to 95%.

Yep, I hope that would be true for this proposal.

> Ken asked "What are we waiting for?" and... I am not sure. The
> PI-partitioning bit - which is key here - should simply "be done" IMHO.
> I say let's get *a single person* to just do it and push a .cs into the
> stream which creates the PIs. I was inclined to get Doug to do it (being
> v3.9 boss and all), but I don't really care as long as someone does it.
> Again, I am not sure how this plays with the latest plans/developments
> (read the packages archive for details).

It still fits in... I can add the partitioning changeset (from iSqueak)
to the restarted 3.9alpha stream as soon as we have a SqueakSource/MC
repository set up for 3.9.

- Doug