On 6/27/05, Doug Way dway@mailcan.com wrote:

...

I'll populate that project shortly.

Sounds good. By the way, does a SqueakSource "Developer" have access to do this initial populating (adding new packages), or do you need to be an Admin?

You don't have to be an admin (and I should probably add some others as admins anyway, no reason to have me be a bottleneck).

However, I just tried to upload the lot and found that there's still something messed up with the squeaksource installation. Argh. I'll look at this again after dinner.

I will set up a new swiki page covering the new process, as a point of reference.

Great.

Avi

Am 27.06.2005 um 18:12 schrieb Alexandre Bergel:

I still think that the mailing list email should be used when answering. Who can configure it ? Or better question, where can it be configured...

You can "configure" that in your mail client. I see you use Mutt. Press L to reply to the list.

See http://www.unicom.com/pw/reply-to-harmful.html

- Bert -

On 6/27/05, Avi Bryant avi.bryant@gmail.com wrote:

On 6/27/05, Avi Bryant avi.bryant@gmail.com wrote:

...

However, I just tried to upload the lot and found that there's still something messed up with the squeaksource installation. Argh. I'll look at this again after dinner.

Just FYI, I'm currently taking the SqueakSource instance down for a bit while I try (again) to sort this out...

Ok, it's back up, and I'm slowly feeding it packages. For now you'll have to access it at port 9090 (http://source.squeakfoundation.org:9090/). Cees, when you get a chance, can you change the ProxyPass lines in the apache config to drop the /seaside/ss part?

Avi

On 6/27/05, Avi Bryant avi.bryant@gmail.com wrote:

So, the first set of packages have been uploaded to the 39a project, and I've placed the image I committed them from to http://beta4.com/~avi/Squeak3.9a-Packages.zip (for now - maybe someone could drop this onto box1?). This should make a good starting point for anyone wanting to start feeding some new changes in.

http://source.squeakfoundation.org/static/Squeak3.9a-Packages.zip - would that be alright as well? ;-)

The apache config is now moderately complex, but I think it all works... FLW...

It makes sense to me. Who set it up ? Let us know once done, I will test.

Alexandre

On Tue, Jun 28, 2005 at 06:21:33PM -0400, Doug Way wrote:

On Mon, 27 Jun 2005 20:55:56 +0200, "Avi Bryant" avi.bryant@gmail.com said:

...

So, the first set of packages have been uploaded to the 39a project, and I've placed the image I committed them from to http://beta4.com/~avi/Squeak3.9a-Packages.zip (for now - maybe someone could drop this onto box1?). This should make a good starting point for anyone wanting to start feeding some new changes in.

Doug, we should start doing whatever needs to be done with the 3.9a update stream now...

Yes, I'm fiddling around with that now.

So, here are the possible updates that would get us started with the 3.9alpha update stream... (Starting with a fresh Squeak3.8-6665-basic image.)

• Update cs #1: Full Monticello changeset. For now, I'm just using

Monticello-ar.245.mcz for this, which is the latest one in the Squeak3.8-Packages iSqueak image, because we want a version that works with MonticelloConfigurations. Or can we just use one of the base Monticello releases such as Monticello-avi.231 ? (Does the Reorg cs depend on a specific Monticello version?)

• Update cs #2: MonticelloConfigurations-ar.17 changeset.

• Update cs #3: Reorganize.3.8-6662.cs (from Andreas)

(Hmmm, what is in the Unnamed cs in the Squeak3.8-Packages image? There's a bunch of changes in there... this is different from the Reorg cs.)

• Update cs #4: The partitioning DoIt cs (from http://source.impara.de

-> Projects -> iSqueak -> wiki).

Note that after I do these steps, the list of packages in the Monticello browser is somewhat different than in the Squeak3.8-Packages image, this may need some cleanup? Not sure where Connectors and some of the other packages came from. See attached screenshot.

Part of this is that we just need to set the release repository (to 'http://source.squeakfoundation.org/39a') for the 35 packages. I assume this can be done with a DoIt. (And eventually we should also set master repositories for these packages...)

Hopefully we can do all of this cleanup through updates, so that people can continue to follow the update stream from a 3.8 final image if they want. Otherwise, we'll have to just tell everyone to download a new 3.9alpha partitioned image, which wouldn't be the end of the world, but it'd be nice to avoid that.

• Doug

On Wednesday, June 29, 2005, at 08:20 AM, Avi Bryant wrote:

On 6/29/05, Doug Way dway@mailcan.com wrote:

...

• Update cs #1: Full Monticello changeset. For now, I'm just using

Monticello-ar.245.mcz for this, which is the latest one in the Squeak3.8-Packages iSqueak image, because we want a version that works with MonticelloConfigurations. Or can we just use one of the base Monticello releases such as Monticello-avi.231 ? (Does the Reorg cs depend on a specific Monticello version?)

We definitely want one of the Impara versions, for the configuration support.

Ok, I'll go with the versions I mentioned for cs #1 and #2 then.

...

• Update cs #4: The partitioning DoIt cs (from http://source.impara.de

-> Projects -> iSqueak -> wiki).

Note that after I do these steps, the list of packages in the Monticello browser is somewhat different than in the Squeak3.8-Packages image, this may need some cleanup? Not sure where Connectors and some of the other packages came from. See attached screenshot.

I think those come from cs #4, right? Do we need to run that? I'm not sure we want all of those automatically created packages in there.

Yes, but I thought cs #4 did all of the other PI-creation as well, so I assume it is necessary. I.e. it creates the Kernel, Collections, etc MC packages from the class categories. Installing MC doesn't do this, does it? The Reorg cs does not do it, it only shuffles class categories around.

Anyway, I can go through the steps again with my image tomorrow (which I don't have with me at the moment), and see where these extra packages are coming from... I should be able to figure this out.

...

Hopefully we can do all of this cleanup through updates, so that people can continue to follow the update stream from a 3.8 final image if they want. Otherwise, we'll have to just tell everyone to download a new 3.9alpha partitioned image, which wouldn't be the end of the world, but it'd be nice to avoid that.

The other thing we'll need to generate will be a changeset that provides the ancestry metadata for all of the initial package versions, so that the MC working copies will be in the same state as they would have been if all the packages had been loaded from the repository (which will take a long time and we don't want to put people through if we don't have to).

Ok, that would be good. Mm, any suggestions on how I should do this? :-) Of course, most of the packages are really the "initial" package versions as you say, so they don't really have any "ancestry", right? E.g. we would have Kernel version 1. But there may be something missing that I should add in any case... as if it were loaded from the repository, I agree.

- Doug

Hi -

Some more comments:

...

...

Note that after I do these steps, the list of packages in the Monticello browser is somewhat different than in the Squeak3.8-Packages image, this may need some cleanup? Not sure where Connectors and some of the other packages came from. See attached screenshot.

I think those come from cs #4, right? Do we need to run that? I'm not sure we want all of those automatically created packages in there.

Depends on whether you wanted to have the system fully packagized or not. This is a key question - I have no interest whatsover in another "partially packaged" system. We have this already.

If you *are* interested in a fully packagized system then you need to make sure you know what every last line of code belongs to. The doIt does that by touching every method in the system and therefore pointing out (!) places that need to be fixed.

That's what this doIt is good for - it's NOT intended to perform proper packagizing it is intended to show which parts of the system are not included in your current package set and therefore need to be dealt with.

Since I did the work a while ago it is quite possible that later changes have done some modifications that I was unaware about when I did the work. I suggest you just reclassify these classes/methods too so that you have a fully packagized system.

Yes, but I thought cs #4 did all of the other PI-creation as well, so I assume it is necessary. I.e. it creates the Kernel, Collections, etc MC packages from the class categories. Installing MC doesn't do this, does it? The Reorg cs does not do it, it only shuffles class categories around.

The doIt "sort of" does this but *not* properly. To do it properly you want both, a repository that the package resides on and a version of the package. Here is how to do this:

a) Send out the Reorg.cs just to get things in order for people out there (this includes any additional modifications that need to be made) b) Do whatever is necessary to get Monticello/MCConfs to the people out there (e.g., put them into the update stream or so) c) Post versions to the 3.9 repository that include all packages which should then be in a basic image d) Post a configuration that people out there get via update stream

If you do this, here is what will happen for a 3.9 user: * They get the ReOrg.cs * They a configuration map * This will force them to download the versions from the repository * Since the repository version will be equivalent to what is in the system, no changes will be made

In other words, the above is a recipe to get the packages to the users at the expense of having to download all of these packages from the repository - which will take some time but it also means that (for once ;-) you know that people have precisely the versions that are in the repository.

Anyway, I can go through the steps again with my image tomorrow (which I don't have with me at the moment), and see where these extra packages are coming from... I should be able to figure this out.

Almost always they are method extensions which originally belonged to some package but where posted with other fixes.

...

...

Hopefully we can do all of this cleanup through updates, so that people can continue to follow the update stream from a 3.8 final image if they want. Otherwise, we'll have to just tell everyone to download a new 3.9alpha partitioned image, which wouldn't be the end of the world, but it'd be nice to avoid that.

See above. The only issue is that everyone will have to download the packages so you guys better get that server in shape ;-)

...

The other thing we'll need to generate will be a changeset that provides the ancestry metadata for all of the initial package versions, so that the MC working copies will be in the same state as they would have been if all the packages had been loaded from the repository (which will take a long time and we don't want to put people through if we don't have to).

Ok, that would be good. Mm, any suggestions on how I should do this? :-) Of course, most of the packages are really the "initial" package versions as you say, so they don't really have any "ancestry", right? E.g. we would have Kernel version 1. But there may be something missing that I should add in any case... as if it were loaded from the repository, I agree.

No! Please, no! Not "as if" - make it the real thing! What is the problem with just downloading the package and installing it? If people don't want to wait they can always just grab a preloaded image.

Cheers, - Andreas

Under the assumption that the placing of code under packages is done simply by creating an MC package for every otherwise-uncategorized method, and therefore is no work at all (I think that's what Andreas says that doIt does, if not, that's not hard to write):

I think its better to have everything in packages, so people don't meet technical problems such as "I changed a bunch of methods, uploaded the packages that changed in MC, but one of my methods didn't appear".

The down side is we might have a couple of dumb packages, but we'll be moving code between packages forever anyway (detangling), so who cares?

Daniel

Avi Bryant wrote:

On 6/30/05, Andreas Raab andreas.raab@gmx.de wrote:

...

Depends on whether you wanted to have the system fully packagized or not. This is a key question - I have no interest whatsover in another "partially packaged" system. We have this already.

I'm certainly with you on having a fully packagized system; the main question I have is whether we want to fully packagize it eagerly or lazily. You've done a great job of getting the bulk of the packaging done, and that may be enough to start with; as long as we have a fully packagized *process*, the rest can come along at a slower rate. That is, if there's no way to submit a FIX or ENH, or get an update from the stream, for code that's not packagized, then the first time we need to actually change one of those loose methods, it will get dealt with - and probably more thoughtfully than if we try to do them all at once now. Another strategy might be to recategorize them to all be in an *orphaned package rather than creating tons of tiny packages, with the rule that when you modify a method in that package, you need to try to move it out (so it should shrink over time).

But that's just my $0.02 - if others are less inclined to punt, and want to do it properly up front, great. It might not turn out to be that much work.

...

No! Please, no! Not "as if" - make it the real thing! What is the problem with just downloading the package and installing it? If people don't want to wait they can always just grab a preloaded image.

Fair enough. I suspect most people will just grab the preloaded one anyway.

Avi

Haven't thought this through. I also lean towards the former, because it seems easier to keep track of.

Daniel

Avi Bryant wrote:

On 6/30/05, Daniel Vainsencher danielv@techunix.technion.ac.il wrote:

...

Under the assumption that the placing of code under packages is done simply by creating an MC package for every otherwise-uncategorized method, and therefore is no work at all (I think that's what Andreas says that doIt does, if not, that's not hard to write):

I think its better to have everything in packages, so people don't meet technical problems such as "I changed a bunch of methods, uploaded the packages that changed in MC, but one of my methods didn't appear".

The down side is we might have a couple of dumb packages, but we'll be moving code between packages forever anyway (detangling), so who cares?

Ok. So is it better to have a single dumb package ("Orphaned") or lots of little dumb packages? I lean towards the former.

Avi

On Wed, 29 Jun 2005 21:41:50 -0700, "Andreas Raab" andreas.raab@gmx.de said:

Hi -

Some more comments:

...

...

...

Note that after I do these steps, the list of packages in the Monticello browser is somewhat different than in the Squeak3.8-Packages image, this may need some cleanup? Not sure where Connectors and some of the other packages came from. See attached screenshot.

I think those come from cs #4, right? Do we need to run that? I'm not sure we want all of those automatically created packages in there.

Depends on whether you wanted to have the system fully packagized or not. This is a key question - I have no interest whatsover in another "partially packaged" system. We have this already.

Agreed 100%... I think we all want a fully packagized system, where every single method in the system belongs to a package. (And preferably, every method belongs to exactly one package, which means the Basic configuration of packages has no overrides, if possible.)

If you *are* interested in a fully packagized system then you need to make sure you know what every last line of code belongs to. The doIt does that by touching every method in the system and therefore pointing out (!) places that need to be fixed.

That's what this doIt is good for - it's NOT intended to perform proper packagizing it is intended to show which parts of the system are not included in your current package set and therefore need to be dealt with.

Actually, we are in better shape than I originally thought. There were a dozen or so extra MC packages that showed up after I executed the DoIt (e.g. "Connectors", "Flexiblevocabularies", "Refactory", etc). However, when browsing each of these packages, they have no code, they just show an empty list of *Extensions and nothing else.

So, my guess is that these were perhaps generated by the DoIt from some extra extension method categories (e.g. "*Connectors") which didn't have any methods in them. Ah, I see a fresh 3.8-6665-basic image has a "*flexiblevocabularies-viewer" method category on Object, which the Reorg cs doesn't do anything with, so that's where they come from... shouldn't be hard to clean that up.

Well, there is one stray package which did have some code in it after I ran the DoIt. A package called "UserObjects" has three classes, CardPlayer55, CardPlayer57, and Player56. Looks like some Etoys prototype classes which don't need to be there. There are no instances of these classes in the 6665-basic image, so I could just add an update cs which deletes them, and deletes the UserObjects class category.

Since I did the work a while ago it is quite possible that later changes have done some modifications that I was unaware about when I did the work. I suggest you just reclassify these classes/methods too so that you have a fully packagized system.

As far as "later changes" go, there are actually only two new updates in 3.8 since your iSqueak Packages image was created, 6663 and 6664. Both of these updates just change a few existing methods, they don't add or reorganize anything. So your Reorg changeset looks like it should be fine as-is, except for the small cleanups I noted above.

...

Yes, but I thought cs #4 did all of the other PI-creation as well, so I assume it is necessary. I.e. it creates the Kernel, Collections, etc MC packages from the class categories. Installing MC doesn't do this, does it? The Reorg cs does not do it, it only shuffles class categories around.

The doIt "sort of" does this but *not* properly. To do it properly you want both, a repository that the package resides on and a version of the package. Here is how to do this:

a) Send out the Reorg.cs just to get things in order for people out there (this includes any additional modifications that need to be made) b) Do whatever is necessary to get Monticello/MCConfs to the people out there (e.g., put them into the update stream or so) c) Post versions to the 3.9 repository that include all packages which should then be in a basic image d) Post a configuration that people out there get via update stream

If you do this, here is what will happen for a 3.9 user:

• They get the ReOrg.cs
• They a configuration map
• This will force them to download the versions from the repository
• Since the repository version will be equivalent to what is in the

system, no changes will be made

In other words, the above is a recipe to get the packages to the users at the expense of having to download all of these packages from the repository - which will take some time but it also means that (for once ;-) you know that people have precisely the versions that are in the repository.

Yeah, that sounds like the way to go, then.

I can put up a big warning dialog before this update, telling people that they should strongly consider just getting a replacement 3.9alpha image from http://ftp.squeak.org/xxx, noting that proceeding with the big package-installing update WILL take longer than downloading a brand-new image. But if they're a glutton for punishment, they can proceed. :)

...

...

...

...

Hopefully we can do all of this cleanup through updates, so that people can continue to follow the update stream from a 3.8 final image if they want. Otherwise, we'll have to just tell everyone to download a new 3.9alpha partitioned image, which wouldn't be the end of the world, but it'd be nice to avoid that.

See above. The only issue is that everyone will have to download the packages so you guys better get that server in shape ;-)

Hopefully with the big warning message, not too many crazy folks will do the big package install. :)

...

... Ok, that would be good. Mm, any suggestions on how I should do this? :-) Of course, most of the packages are really the "initial" package versions as you say, so they don't really have any "ancestry", right? E.g. we would have Kernel version 1. But there may be something missing that I should add in any case... as if it were loaded from the repository, I agree.

No! Please, no! Not "as if" - make it the real thing! What is the problem with just downloading the package and installing it? If people don't want to wait they can always just grab a preloaded image.

Yes, sounds good to me, I wasn't looking forward to "faking" this ancestry.

I'll see if I can get these new updates added to the 3.9alpha stream tonight.

- Doug

On Jun 30, 2005, at 12:25 PM, Doug Way wrote:

I'll see if I can get these new updates added to the 3.9alpha stream tonight.

Right now I'm loading all the packages from the 39a repository into a test image, so I can put together an MCConfiguration. However, the Morphic package causes a problem when I load it... there are a few changes in the code related to the reorg. (Specifically, the reorg moved some essential methods such as Morph>>handlerForMouseDown: from a "*StandardYellow..." category to a plain "standardyellow..." category, so that they're no longer extensions.)

Avi, were the packages you put in the 39a repository created without installing the Reorg changeset first? That would explain the difference. I may have to put some updated versions of the packages in the 39a repository to be exactly consistent with the source in the 3.8+reorg image.

- Doug

On Jul 2, 2005, at 5:48 AM, Avi Bryant wrote:

On Jul 2, 2005, at 6:22 AM, Doug Way wrote:

...

Right now I'm loading all the packages from the 39a repository into a test image, so I can put together an MCConfiguration. However, the Morphic package causes a problem when I load it... there are a few changes in the code related to the reorg. (Specifically, the reorg moved some essential methods such as Morph>>handlerForMouseDown: from a "*StandardYellow..." category to a plain "standardyellow..." category, so that they're no longer extensions.)

Avi, were the packages you put in the 39a repository created without installing the Reorg changeset first? That would explain the difference. I may have to put some updated versions of the packages in the 39a repository to be exactly consistent with the source in the 3.8+reorg image.

Yes, I think that's right - so, good, commit them; this will give the system a little more of a workout.

Alright, I started committing them, I have an image set up where I could commit them all.

Well, I just committed one so far, Balloon-dew.1.mcz. Since it was a new package created from scratch (from my own partitioning DoIt), it starts at version 1. This is good, I think, since it really has no ancestry. All the packages could start at version 1, except for Monticello & SqueakMap, which do have an MC ancestry.

The only problem is that you have a version 4 (Balloon-avi.4.mcz) out there in the 39a repository, and SqueakSource thinks that is the "latest" one. Should I try to create mine as a version 5 instead? (Not sure offhand how to do that.) Or is there a way to delete your packages?

- Doug

p.s. for (mostly my own) reference, here's my list of steps that I'm using for all of this:

1. Generate packages for 39a repository:

- Start with 3.8-6665-basic image, rename - Install Reorg cs - Install my CleanupUserObjects cs (removes UserObjects & its etoy-generated classes) - Install latest MC.261 + MCConfigs.30 from Bert - Run my package-creation DoIt, which creates exactly 35 packages in the MC browser: "Create MC packages for all system category prefixes." (SystemOrganization categories collect: [:ea | (ea findTokens: $-) first]) asSet do: [:prefix | PackageInfo registerPackageName: prefix. MCWorkingCopy forPackage: (MCPackage new name: prefix)]. - Add 39a repository in MC (with my id/pw), add to all packages in MC. - Save all 35 packages to the 39a repository.

2. Post initial updates to the 3.9alpha update stream:

- In updates.list, rename previous #3.9alpha stream tag to #3.9alphaold, to keep it around for reference. Add new #3.9alpha tag at the end to start it over. - Add Reorg cs as first update. (Hmm, may not be entirely necessary, but probably a good idea so that the packages installed later have identical code.) - Add my CleanupUserObjects cs as the 2nd update. - Add two changesets based on Bert's MC & MCConfigurations installs, as 3rd & 4th updates.

3. Create MCConfiguration changeset to be added to the update stream:

- Start with 3.8-6665-basic image, rename - Install Reorg cs - Install MC + MCConfigs from Bert - Load all 35 latest packages from the 39a repository (the ones I saved there previously) via MC, one by one. Before each load, check the "changes" to make sure there are no changes, i.e. the package and the code in the image should be identical. - Open the MCConfigurations browser, add all 35 packages, in the preferred load order. For this first MCConfiguration, I'm not sure the load order matters all that much, since the loaded code should be identical to what's already in the image. I guess I will start with the "root" packages first (Kernel, Collections, etc) and add the outer ones later in the list. - Post the configuration to the 3.9alpha update stream (as the 5th update), see how it goes.

4. Test the 3.9alpha stream by updating from a fresh image!

On Jul 3, 2005, at 5:03 AM, stéphane ducasse wrote:

Hi doug

I just registered to the 3.9 squeaksource. Tell me how I can help. Where can I find a 3.9 packaged image?

I'm still working on it, see my other post. I realize I'm the bottleneck here, but I have some time this weekend and I will get it done today or tomorrow.

In the meantime, I added you as a developer on the 3.9a project. (And also Marcus Denker.) The current list of developers is roughly the group which had access to the update stream previously. We probably don't need a lot of developers on the 3.9a project... keep in mind that the bulk of development should be happening in the individual packages in their master repositories, in theory. We will see how it all works out. :)

- Doug

Just a question is the configuration file versioned too because it seems important to be able to load back old configurations?

Stef

On 5 juil. 05, at 11:26, Bert Freudenberg wrote:

Am 04.07.2005 um 08:29 schrieb stéphane ducasse:

...

Just a question is the configuration file versioned too because it seems important to be able to load back old configurations?

In the current hybrid stream/mc scheme, there is no separate configuration file. All configurations will appear as a changeset in the update stream. You can use the changesorter to see older configurations.

OK Is there a reason? because publishing the MC configurations also as a package is useful.

Stef

• Bert -

• start with a fresh 3.8-6665-basic image (save a copy)
• 'SystemVersion setVersion' to 'Squeak3.9alpha'
• load updates

When I do that, I get a rollback "Error: cannot remove non-empty category". Change 6672CleanupUserObjects-dew yields this. Am I the only one ?

The image I used is a fresh 6665.

Alexandre

I'm going to pack it in for the day, but hopefully this won't take too long to fix tomorrow. Then I can post a working image with the updates already loaded.

• Doug

p.s. these are the updates:

6670New39StreamWarning-dew.cs 6671Reorganize39-6665-ar.cs 6672CleanupUserObjects-dew.cs 6673Monticello-bf.cs 6674MonticelloConfigs-bf.cs 6675InstallPkgsWarning-dew.cs 6676InstallMCPackages-dew.cs

...

When I do that, I get a rollback "Error: cannot remove non-empty category". Change 6672CleanupUserObjects-dew yields this. Am I the only one ?

The image I used is a fresh 6665.

It works fine.

I have some changes related to the changeset notification. how should I proceed to submit it ?

I loaded my changeset, and the System is modified. Shall I save the package System?

As far as I understood, people willing to submit fixes, have to do it using changeset, and harvester are the only ones who can update the 39a.

If what I understood is correct, I will contribute to the swiki page Doug started once I am back in bern next week.

Cheers, Alexandre

On 5 juil. 05, at 22:37, Doug Way wrote:

Anyway, ideally we should have a master System package repository set up somewhere. Could be at source.sqf.org or squeaksource.com or somewhere else. As one of the co-maintainers, you would simply commit your changes to the System package there. Periodically, we (the v3.9 team) would add the latest version of the System package to the 39a repository as well, and then we'd have your changes.

(The cool thing about the master vs release repository separation is that if the maintainers of the System package don't do their job, someone else could create a competing System package in a different repository, and the v3.9 team could simply switch to using that version instead.)

But it seems that for now this can be really difficult for refactorings that are crosscutting several packages since you will have to look in different locations. So this can be a real pain. Or will I publish in another repository so this means that we will create a lot of hidden dependencies because even if I work regularly but another master is on holiday then the v3.9 team can be just out of the loop to get my changes in.

Stef

On Tue, 5 Jul 2005 23:05:17 +0200, "stéphane ducasse" ducasse@iam.unibe.ch said:

...

In the short term, one thing we could do is simply make changes to the System package and other core packages directly in the 39a repository, since we don't have master repositories for these packages yet. I hesitate to suggest this, though, as this is still too centralized a model. But if we have trouble finding package maintainers, we may end up having to do this for some packages in the short term.

My impression is that we should go slowly and do not go crazy and decentralized too fast. For well identified packages such as MC this is really ok but for the others this can be really a mess as I mentioned.

I think I agree. Maybe for some of the well-defined "outer" packages such as Balloon, Movies, Speech, etc., we should try to find maintainers and get someone to set up master repositories soon for those, at least. (Hm, some of those maybe don't even need to be in Basic, they could just be part of Full.)

But some of the core packages haven't really been properly detangled yet, so we may have a lot of changes which cut across several packages. Maybe these core packages (Kernel, Collections, Morphic, etc) should be maintained in the 39a repository for the short term until they are better detangled. Certainly the detangling work itself will definitely cut across multiple packages and would be easier to do directly in the 39a repository. (Although the work of the Morphic Splitters team is obviously relevant here, I'll have to see what progress they made.) Comments from Daniel and any other detangling experts? :)

Actually, both the Morphic Splitters and the Toolbuilder work involve some detangling, and we'd probably want to look at incorporating both of those earlier rather than later. (At a certain point we just have to pick which order these are incorporated... e.g. 1. Morphic Splitters 2. Toolbuilder refactoring 3. previous 3.9alpha changes... whichever goes first has the least amount of merging to do.)

- Doug

But some of the core packages haven't really been properly detangled yet, so we may have a lot of changes which cut across several packages.

I agree. Better package separation is probably the most important thing to do right now - simply because otherwise you'll end up with inappropriate changes to (say) Kernel because of some Morphic change. It would be nice if package boundaries could be defined in such a way that we know and can proove the package dependencies and use that as a guide line for how to set up the boundaries.

Maybe these core packages (Kernel, Collections, Morphic, etc) should be maintained in the 39a repository for the short term until they are better detangled. Certainly the detangling work itself will definitely cut across multiple packages and would be easier to do directly in the 39a repository. (Although the work of the Morphic Splitters team is obviously relevant here, I'll have to see what progress they made.) Comments from Daniel and any other detangling experts? :)

Keep in mind that MC is fairly good at merging (and where it isn't it needs fixing ;-) so we ought to be able to take the results of various projects and simply merge them in.

Actually, both the Morphic Splitters and the Toolbuilder work involve some detangling, and we'd probably want to look at incorporating both of those earlier rather than later. (At a certain point we just have to pick which order these are incorporated... e.g. 1. Morphic Splitters 2. Toolbuilder refactoring 3. previous 3.9alpha changes... whichever goes first has the least amount of merging to do.)

As far as I can tell 1. and 2. don't overlap at all (Brian could say more about this). 3. should probably go last since it has the finest granularity and therefore is easiest to merge.

Cheers, - Andreas

There's a Submissions repository world-writable, on source.sqf.org.

We can post things there for people to look at, and occasionally merge it into 3.9a. So one way to begin would be to start creating versions on Submissions that include the patches that have accumulated for 3.9a before, merging them into the 39a repository versions as people think is safe.

Might be a good idea to wait with creating a master repository for each package until its clearer who are the maintainers, and the borders between packages stabilize. In the mean time, Submissions could be used for everything.

Daniel

Doug Way wrote:

On Tue, 5 Jul 2005 10:20:58 +0200, "Alexandre Bergel" bergel@iam.unibe.ch said:

...

It works fine.

I have some changes related to the changeset notification. how should I proceed to submit it ?

I loaded my changeset, and the System is modified. Shall I save the package System?

As far as I understood, people willing to submit fixes, have to do it using changeset, and harvester are the only ones who can update the 39a.

Ok, here is where things get interesting. :)

Basically, most of the old rules are now gone. You won't have one group of harvesters controlling everything (so that everyone else has to send them changesets) anymore.

We need to get small groups of 2-6 maintainers for all of the 35 packages making up the base image. Easier said than done, of course, but we need to make a stab at it.

In this case, it sounds like all of your changes are in the System package? (When you make your changes, does only the System package have a "*" by it in the MC Browser?)

If that is the case, since you are doing significant work in the System package, I would suggest that you be one of the co-maintainers of the master System package. Various folks at Berne have made significant changes to System in the last few years (KCP team etc), so it makes sense for you or someone else there to be one of the co-maintainers.

With an important package like System, it's probably good to have at least 3 maintainers, so that the group is self-sustaining if somebody leaves, they can review each other's changes, etc. One of these people would be the "lead" maintainer. More trivial packages such as VersionNumber could probably just have one maintainer. (This is really a topic for a whole other email.)

Anyway, ideally we should have a master System package repository set up somewhere. Could be at source.sqf.org or squeaksource.com or somewhere else. As one of the co-maintainers, you would simply commit your changes to the System package there. Periodically, we (the v3.9 team) would add the latest version of the System package to the 39a repository as well, and then we'd have your changes.

(The cool thing about the master vs release repository separation is that if the maintainers of the System package don't do their job, someone else could create a competing System package in a different repository, and the v3.9 team could simply switch to using that version instead.)

So, the master repository vs release repository aspect is a little bit complicated, but it seems like the only real way to move to a system of truly distributed package development. We do already have master repositories for some packages, such as MC, MCConfigurations, SM-Base, etc. Now we need them for all the other packages.

In the short term, one thing we could do is simply make changes to the System package and other core packages directly in the 39a repository, since we don't have master repositories for these packages yet. I hesitate to suggest this, though, as this is still too centralized a model. But if we have trouble finding package maintainers, we may end up having to do this for some packages in the short term.

It is still true that an outsider submitting a bug fix should submit a changeset via Mantis or whatever current process. The package maintainers would be checking Mantis periodically for fixes from the community, and incorporating them into their package.

...

If what I understood is correct, I will contribute to the swiki page Doug started once I am back in bern next week.

So, you got that? ;-)

For now, I'm not sure if you should go ahead and make the change in the 39a System package, or if you should wait until we do a real maintainership drive, and create all the remaining master packages. Thoughts from anyone else?

• Doug

Ah, you do see my mails, just a matter of timing.

Doug Way wrote:

On Wed, 06 Jul 2005 00:16:43 +0300, "Daniel Vainsencher" danielv@techunix.technion.ac.il said:

...

There's a Submissions repository world-writable, on source.sqf.org.

We can post things there for people to look at, and occasionally merge it into 3.9a. So one way to begin would be to start creating versions on Submissions that include the patches that have accumulated for 3.9a before, merging them into the 39a repository versions as people think is safe.

The Submissions repository seems like it might be an extra step which is not entirely necessary... we already have these changes stored in the old update stream, or on Mantis, etc.

[snip]

One upside to the Submissions repository is that the UI is available in Squeak via MC, whereas Mantis changesets are not.

I don't think of it as "show up in Squeak via MC". I think of it as - can be trivially accessed by Squeak code, in a well modeled manner, using MC. On top of that we can build various things, better UI being just one. For example, a test server that runs a test suite on every submitted version, catching regressions before they get into the update stream.

I guess one possibility is still using Mantis to track bugs & fixes, but posting the code in the Submissions repository instead of to Mantis.

Yup. Given a common repository for each package, the mantis entry needs only to name the specific version. And as Bert suggested, we could eventually move bug tracking functionality into Squeak.

One downside is that the repository could easily get cluttered with hundreds of unordered submissions.

I think that's a tools issue. That repositories accumulate dead end versions is natural. If this repository does it faster, and pushes SqueakSource and MC to improve how they deal with this, or specialized UIs to appear, that's a good thing.

It seems more relevant as a possibly new fix/enh submissions process, which is currently done in Mantis, and is partly/mostly handled by the Janitors group. You might want to bring up the idea on that list, although I guess it's worth pondering here for a bit.

Sure. Seems to me that we want the process to be as seamless as possible, while allowing maintainers control.

So I propose for each package, things get merged into 39a from its master repository, which can be Submissions if there's nothing else, or the maintainers repository otherwise. We make sure that in the image, each package includes its master repository, so things get submitted to the right place. If the maintainer sees something interesting for his package that got to Submissions by mistake, he merges it (or not), and again, nobody else needs to do anything special - just merge from the maintainer every so often.

And from everyones point of view (Joe R. Fixit, Maintainer, update stream writer (are those still called harvesters?)), everything is quite uniform. I get my code for a package from one predefined repository, and merge and save what I think is right into another.

Daniel Vainsencher

Sure, I will work on it next week, once we're back.

Cheers, Alexandre

On Tue, Jul 05, 2005 at 04:37:27PM -0400, Doug Way wrote:

On Tue, 5 Jul 2005 10:20:58 +0200, "Alexandre Bergel" bergel@iam.unibe.ch said:

...

It works fine.

I have some changes related to the changeset notification. how should I proceed to submit it ?

I loaded my changeset, and the System is modified. Shall I save the package System?

As far as I understood, people willing to submit fixes, have to do it using changeset, and harvester are the only ones who can update the 39a.

Ok, here is where things get interesting. :)

Basically, most of the old rules are now gone. You won't have one group of harvesters controlling everything (so that everyone else has to send them changesets) anymore.

We need to get small groups of 2-6 maintainers for all of the 35 packages making up the base image. Easier said than done, of course, but we need to make a stab at it.

In this case, it sounds like all of your changes are in the System package? (When you make your changes, does only the System package have a "*" by it in the MC Browser?)

If that is the case, since you are doing significant work in the System package, I would suggest that you be one of the co-maintainers of the master System package. Various folks at Berne have made significant changes to System in the last few years (KCP team etc), so it makes sense for you or someone else there to be one of the co-maintainers.

With an important package like System, it's probably good to have at least 3 maintainers, so that the group is self-sustaining if somebody leaves, they can review each other's changes, etc. One of these people would be the "lead" maintainer. More trivial packages such as VersionNumber could probably just have one maintainer. (This is really a topic for a whole other email.)

Anyway, ideally we should have a master System package repository set up somewhere. Could be at source.sqf.org or squeaksource.com or somewhere else. As one of the co-maintainers, you would simply commit your changes to the System package there. Periodically, we (the v3.9 team) would add the latest version of the System package to the 39a repository as well, and then we'd have your changes.

(The cool thing about the master vs release repository separation is that if the maintainers of the System package don't do their job, someone else could create a competing System package in a different repository, and the v3.9 team could simply switch to using that version instead.)

So, the master repository vs release repository aspect is a little bit complicated, but it seems like the only real way to move to a system of truly distributed package development. We do already have master repositories for some packages, such as MC, MCConfigurations, SM-Base, etc. Now we need them for all the other packages.

In the short term, one thing we could do is simply make changes to the System package and other core packages directly in the 39a repository, since we don't have master repositories for these packages yet. I hesitate to suggest this, though, as this is still too centralized a model. But if we have trouble finding package maintainers, we may end up having to do this for some packages in the short term.

It is still true that an outsider submitting a bug fix should submit a changeset via Mantis or whatever current process. The package maintainers would be checking Mantis periodically for fixes from the community, and incorporating them into their package.

...

If what I understood is correct, I will contribute to the swiki page Doug started once I am back in bern next week.

So, you got that? ;-)

For now, I'm not sure if you should go ahead and make the change in the 39a System package, or if you should wait until we do a real maintainership drive, and create all the remaining master packages. Thoughts from anyone else?

• Doug

On Mon, 4 Jul 2005, Doug Way wrote:

On Jul 4, 2005, at 1:09 AM, Doug Way wrote:

...

... However, there's a problem with the MCConfiguration install. It generally works and gets most of the way through it, but it hits an error trying to install the SMBase-dew.64 package. Looking at the debugger, it looks like it's trying to install an .mcd diff file for this. Ah, maybe the base version has to be in the same repository? (which would be SMBase-gk.63)

Unfortunately, I can't look into this right now.

But there might be a bug in SqueakSource - the MCConf upgrade process tries to download the .mcd first (because that's much more efficient), and if that fails, it downloads the .mcz. So the server should return an error if it cannot build the diff. However, it might be that I never tried that because we usually have all package versions in the repository.

If that is not the problem, it might work if you just add another repository to the workingcopy which has the base version. But I'm not quite sure about that.

Forgot to attach the exception stack earlier... attached below.

The string parameter passed further down the stack in the method MCHttpRepository(MCFileBasedRepository)>>loadVersionFromFileNamed: is 'SMBase-dew.64(gk.63).mcd'.

I'll be looking at this problem shortly.

• Doug

---

4 July 2005 12:22:24 pm

VM: Mac OS - a SmalltalkImage Image: Squeak3.9alpha [latest update: #6675]

SecurityManager state: Restricted: false FileAccess: true SocketAccess: true Working Dir Macintosh HD:Users:dway:Squeak:Squeak 3.8 Trusted Dir Macintosh HD:Users:dway:Squeak:Squeak 3.8 Untrusted Dir foobar:tooBar:forSqueak:bogus

ZipArchive(Object)>>error: Receiver: a ZipArchive Arguments and temporary variables: aString: 'can''t find EOCD position' Receiver's instance variables: members: an OrderedCollection() centralDirectorySize: nil centralDirectoryOffsetWRTStartingDiskNumber: nil zipFileComment: '' writeCentralDirectoryOffset: 0 writeEOCDOffset: 0

ZipArchive>>readFrom: Receiver: a ZipArchive Arguments and temporary variables: aStreamOrFileName: a RWBinaryOrTextStream a ByteArray(60 33 68 79 67 84 89 80 6...etc... stream: a RWBinaryOrTextStream a ByteArray(60 33 68 79 67 84 89 80 69 32 72 84 ...etc... name: 'a stream' eocdPosition: 0 Receiver's instance variables: members: an OrderedCollection() centralDirectorySize: nil centralDirectoryOffsetWRTStartingDiskNumber: nil zipFileComment: '' writeCentralDirectoryOffset: 0 writeEOCDOffset: 0

MCMcdReader(MCMczReader)>>zip Receiver: a MCMcdReader Arguments and temporary variables:

Receiver's instance variables: stream: a RWBinaryOrTextStream a ByteArray(60 33 68 79 67 84 89 80 69 32 72 84 ...etc... package: nil info: nil definitions: nil dependencies: nil stepChildren: nil zip: a ZipArchive infoCache: nil baseInfo: nil patch: nil

MCMcdReader(MCMczReader)>>parseMember: Receiver: a MCMcdReader Arguments and temporary variables: fileName: 'package' tokens: nil Receiver's instance variables: stream: a RWBinaryOrTextStream a ByteArray(60 33 68 79 67 84 89 80 69 32 72 84 ...etc... package: nil info: nil definitions: nil dependencies: nil stepChildren: nil zip: a ZipArchive infoCache: nil baseInfo: nil patch: nil

--- The full stack --- ZipArchive(Object)>>error: ZipArchive>>readFrom: MCMcdReader(MCMczReader)>>zip MCMcdReader(MCMczReader)>>parseMember:

---

MCMcdReader(MCMczReader)>>loadPackage MCMcdReader(MCVersionReader)>>package MCMcdReader>>basicVersion MCMcdReader(MCVersionReader)>>version [] in MCHttpRepository(MCFileBasedRepository)>>loadVersionFromFileNamed: {[:r | r version]} [] in MCHttpRepository(MCFileBasedRepository)>>versionReaderForFileNamed:do: {[:class | aBlock value: (class on: s fileName: aString)]} MCMcdReader class(Object)>>ifNotNilDo: [] in MCHttpRepository(MCFileBasedRepository)>>versionReaderForFileNamed:do: {[:s | (MCVersionReader readerClassForFileNamed: aString) ifNotNilDo: [:cla...]} MCHttpRepository>>readStreamForFileNamed:do: MCHttpRepository(MCFileBasedRepository)>>versionReaderForFileNamed:do: MCHttpRepository(MCFileBasedRepository)>>loadVersionFromFileNamed: [] in MCHttpRepository(MCFileBasedRepository)>>versionFromFileNamed: {[self loadVersionFromFileNamed: aString]} Dictionary>>at:ifAbsent: MCHttpRepository(MCFileBasedRepository)>>versionFromFileNamed: MCConfiguration>>versionNamed:for:from: [] in MCConfiguration>>depsSatisfying:versionDo:displayingProgress: {[:dep | ver := dep versionInfo name. repo := repoMap at: ver ifAbs...]} [] in OrderedCollection(SequenceableCollection)>>do:displayingProgress: {[:each :i | bar value: i. aBlock value: each]} OrderedCollection(SequenceableCollection)>>withIndexDo: [] in OrderedCollection(SequenceableCollection)>>do:displayingProgress: {[:bar | self withIndexDo: [:each :i | bar value: i. aBlock value: e...]} [] in ProgressInitiationException>>defaultMorphicAction {[result := workBlock value: progress]} BlockContext>>ensure: ProgressInitiationException>>defaultMorphicAction ProgressInitiationException>>defaultAction UndefinedObject>>handleSignal: MethodContext(ContextPart)>>handleSignal: MethodContext(ContextPart)>>handleSignal: ProgressInitiationException(Exception)>>signal ProgressInitiationException>>display:at:from:to:during: ProgressInitiationException class>>display:at:from:to:during: ByteString(String)>>displayProgressAt:from:to:during: OrderedCollection(SequenceableCollection)>>do:displayingProgress: MCConfiguration>>depsSatisfying:versionDo:displayingProgress: ...etc...

Am 05.07.2005 um 06:08 schrieb Doug Way:

On Jul 4, 2005, at 3:47 PM, bert@impara.de wrote:

...

...

On Jul 4, 2005, at 1:09 AM, Doug Way wrote:

...

... However, there's a problem with the MCConfiguration install. It generally works and gets most of the way through it, but it hits an error trying to install the SMBase-dew.64 package. ...

Unfortunately, I can't look into this right now.

But there might be a bug in SqueakSource - the MCConf upgrade process tries to download the .mcd first (because that's much more efficient), and if that fails, it downloads the .mcz. So the server should return an error if it cannot build the diff. However, it might be that I never tried that because we usually have all package versions in the repository. ...

Thanks Bert, that was enough for me to go on, at least. I came up with a fix in MCConfiguration>>versionNamed:for:from: so that it checks if the base version is in the repository first. It may be better to fix in SqueakSource, but this seems like a reasonable fix for now.

And of course, I used the new process to commit the fix in 39a. :-) So, there is a new MonticelloConfigurations-dew.32 in the 39a repository at source.squeakfoundation.org with the fix. I also changed the MCConfig update #6676 to include this new version. (Since it was broken before, I just overwrote it in place.) You can grab the change for the "master" MCConfigurations repository (at Impara?) if it looks good to you.

Ok. I'll be back to Squeak work next week so I can look.

So anyway, the new 3.9alpha update stream is working! It installs SMBase and everything else just fine.

One catch, though... the resulting image is dramatically larger than the original. 3.8-6665 was 11.9MB, the new 3.9a-6676 is 27.7MB... oof. Not sure if this is a known problem or if it's something that can be easily cleaned up, I've haven't looked into it yet. The .changes file is the same size.

In any case, I put this new 3.9a-6676 image + changes in a zip file at http://update.squeakfoundation.org/external/Squeak3.9a-6676.zip for now. Check it out. We may want to hold off on making this more publicly available until we look into why the image is so large.

Make sure you flush the Monticello caches (there is a menu option for that in the MC browser's repository list menu).

- Bert -

Am 05.07.2005 um 16:51 schrieb Doug Way:

On Tue, 5 Jul 2005 11:31:26 +0200, "Bert Freudenberg" bert@impara.de said:

...

Am 05.07.2005 um 06:08 schrieb Doug Way:

...

One catch, though... the resulting image is dramatically larger than the original. 3.8-6665 was 11.9MB, the new 3.9a-6676 is 27.7MB... oof. Not sure if this is a known problem or if it's something that can be easily cleaned up, I've haven't looked into it yet. ...

Make sure you flush the Monticello caches (there is a menu option for that in the MC browser's repository list menu).

Ahh, yes, when I do this, I get a little dialog saying "35 versions flushed, 30,322,492 bytes reclaimed". Whew. :) The saved image is back down to about 13MB.

Well, the size guestimate might not be quite correct, but it adds up to quite a lot memory.\

I'll upload this smaller image/changes pair as a zip file later tonight.

I suppose I could even add an update to do this cache flushing, just for this initial loading of all packages.

Yes ... we might also add a preference to flush the caches before saving the image ... Let's hear Avi's opinion on that.

- Bert -

On Tue, 5 Jul 2005 22:53:17 +0200, "stéphane ducasse" ducasse@iam.unibe.ch said:

Hi doug

I got a look at the image. Thanks for the work. Now how do we proceed?

• Do we try to shrink a bit the packages because some of them are huge?
• Do we try to get used to it by treating the pending fixes first?

Probably we should just work on a few pending fixes first, so we get used to the new process.

• to create a new package should I create a project on

source.squeakfoundation.org? I guess so.

Would this be for one of the 35 packages which doesn't have a master repository yet? (such as System or Kernel) We might want to wait on that for a bit until we know who will be the maintainers.

• I gave a try I loaded 6517KCPRemoveComponent1.cs and republish Morphic. - I got an error HTTP unauthorized 401 at the end of the publish

Ah, here you just need to click on the 39a repository in the right pane of the MC browser and "edit repository info" to include your source.sqf.org userid and password. Since you are set as a developer on the 39a project, you should then have access to save/publish a new version.

I guess we can do a little bit of testing like this but we shouldn't go too crazy with a lot of changes directly in the 39a repository right away. As an admin, it looks like I have access to delete versions, so if there's an obvious mistake, we can always clean it up.

• I removed from the 6517KCPRemoveComponent1 from update.list (this

one a really simple clean that I did and that was in the 3.9 alpha stream).

Actually you don't have to worry about editing any of that old stuff in updates.list now, I marked that whole section of the file as "#Squeak3.9alphaold" so all of those 6517 etc updates are now orphaned.

• then does it means that I should recreate a new cs configuration

changing Morphic-dew.1 into Morphic-sd.2?

Actually no, we only need to do that if we have a DoIt which has to go in the image as an update. See Andreas' original email listed on this page: http://minnow.cc.gatech.edu/squeak/5718

We should just simply load all of the latest packages after checking the update stream. Probably we should directly tweak the "update code from server" command so that this is done automatically after checking the update stream. Um, Andreas mentions that the latest packages should be loaded in a "well-defined order", but I'm not sure where that is defined or how it is done. Is the most recent MCConfiguration load order still available?

- Doug

We should just simply load all of the latest packages after checking the update stream. Probably we should directly tweak the "update code from server" command so that this is done automatically after checking the update stream. Um, Andreas mentions that the latest packages should be loaded in a "well-defined order", but I'm not sure where that is defined or how it is done. Is the most recent MCConfiguration load order still available?

In Tweak we have basically a fixed load order (==configuration) which originates from the configuration used in the original build. But the main reason for wanting to have a well-defined load order is that typically, if there are sets of cross-package changes, they follow already established dependencies. Meaning that if I change a base class to include a new property and then some higher-level code to use it the load-order simply ensures that by default we load the dependent package version first. It may work without the load order, too, but the way it is right now is just clean and simple.

Cheers, - Andreas

Hi Doug -

...

In Tweak we have basically a fixed load order (==configuration) which originates from the configuration used in the original build. But the main reason for wanting to have a well-defined load order is that typically, if there are sets of cross-package changes, they follow already established dependencies.

Yes, makes sense. That should reduce the need for special update changesets to load packages in a special order.

Yes. That's another implicit advantage - most of the time stuff just "works" by loading the latest package versions.

Any suggestions on how I should implement this? I guess we could just have a method defined somewhere (in the load-updates code) which has a hardcoded, ordered list of the 35 package names. I guess this list won't change very often. (The 3.9a-6676 image doesn't seem to be hanging onto the MCConfiguration instance that was loaded in update #6676, otherwise I might try to use that.)

What we've done so far is to have a global which stores the latest version of a configuration and then just works issuing a "updateFromRepositories". It's very simple stuff - if you are curious check out CProjectBuilder>>loadUpdates and updateFromRepositories.

...

Meaning that if I change a base class to include a new property and then some higher-level code to use it the load-order simply ensures that by default we load the dependent package version first.

The first half of your sentence makes it sound like the prerequisite/base packages (e.g. Kernel) should be loaded first, which I think sounds right. Then the outer ("dependent") packages later, not earlier, right? ("dependent" is the opposite of "prerequisite" AFAIK.)

Yes.

Cheers, - Andreas

On Jul 7, 2005, at 6:16 AM, Bert Freudenberg wrote:

Am 07.07.2005 um 08:35 schrieb Andreas Raab:

...

...

Any suggestions on how I should implement this? I guess we could just have a method defined somewhere (in the load-updates code) which has a hardcoded, ordered list of the 35 package names. I guess this list won't change very often. (The 3.9a-6676 image doesn't seem to be hanging onto the MCConfiguration instance that was loaded in update #6676, otherwise I might try to use that.)

That "hardcoded, ordered list of the 35 package names" would just be like the literal configuration that you issue as changeset (without the #upgrade call), but put into a method. It is precisely that literal representation why the configuration is as simple as it is now (compared to a full-blown MCVersion). It just defines a load order for a few packages, that's all :)

Right, sounds good.

I haven't tried adding this yet to the update stream.

This may become obvious when I look into it, but I assume this MC configuration will load the very latest versions of each package, not necessarily the specified versions? (That's what I was looking for... a simple "load order" configuration and not a configuration that only loaded specific versions.) I think that's what the #upgrade does if I remember right, but I'll have to try it.

I assume the CProjectBuilder class below is in a particular Tweak package. Meaning that if you wanted to change the list of packages in #defaultConfiguration, the #defaultConfiguration method in that package would be updated. (In other words, that particular package would sort of store the configuration.) But I guess it wouldn't change that often, if it was simply used for the load order (and didn't have to be updated every time one package in the list bumped versions).

- Doug

...

What we've done so far is to have a global which stores the latest version of a configuration and then just works issuing a "updateFromRepositories". It's very simple stuff - if you are curious check out CProjectBuilder>>loadUpdates and updateFromRepositories.

Yep, this works nicely in Tweak. Pasting the methods below for easier reference. I stripped out the progress display and logging.

loadUpdates TweakUpdateStreamManager loadUpdates. "from update stream" self updateFromRepositories.

updateFromRepositories | map | map := self defaultConfiguration. [map updateFromRepositories. map upgrade. DefaultConfiguration == nil] whileTrue. "repeat if we have a new map"

defaultConfiguration ^DefaultConfiguration ifNil: [ DefaultConfiguration := MCConfiguration fromArray: #( ... )]

There is a hook to set DefaultConfiguration to nil whenever the #defaultConfiguration method is recompiled (but this relies on Tweak semantics). Alternatively, you might have a class-side #initialize method that contains the new configuration, so it would automatically get overwritten after an update, and you test for map identity in #updateFromRepositories rather than testing for nil.

The call to #updateFromRepositories in #loadUpdates might possibly be guarded by a "bleeding edge" preference. That way, normal users only would get the "stable" configuratons explicitly posted to the stream, while the system developers would get all the latest packages from the repository.

Alright, that all makes sense below. I'm trying to create a #loadLatestMCPackages method along the lines of what you describe.

Earlier this week, as a test, I tried to run MCConfiguration>>updateFromRepositories on the MCConfiguration defined in update #6676. It hung, and I didn't have time to look into it further earlier, but I have a bit more time now. :)

So, the main problem here is that it tries to load an .mcd file for the Kernel package: Kernel-md.5(dew.1).mcd, which doesn't exist. Both the original Kernel-dew. and the newer Kernel-md.5 versions are in the 39a repository. (Marcus has been porting a few fixes to the new 3.9a packages, thus the new versions... cool stuff!)

There are two sub-problems here:

1. It just hangs when it tries to get a file which doesn't exist on SqueakSource. It should return some sort of not found error instead. (You mention this exact problem in your 7/12 email... apparently the impara.de SqueakSource server doesn't have this problem.) Help from Avi/Cees or a SqueakSource guy? :-)

2. The .mcd file does not exist. How is this intended to work? A. Is the SqueakSource server supposed to generate the .mcd file on the fly, or maybe upon the saving of a new version? (Makes sense, that's a lot less bandwidth for these big packages.) B. Should the MCConfig code in the image not be trying to download .mcd files? C. Do we need to be saving .mcd files and not .mcz when we save new versions to the repository?

(I'm pretty sure the answer isn't C, that would be dumb. I'm guessing A.)

Anyway, once we get this working, I'll create a #loadLatestMCPackages method and incorporate that as part of the Utilities>>updateFromServer code, and issue that as a new update. Then we should be ready to go!

(I'll probably do something like the class-side #initialize containing the configuration, so that it gets overwritten after an update, and then test it for equality, which you mentioned earlier.)

Mmm, I may just put this "configuration" in the System package for now (where the rest of the updating code is), since it's really just a non-version-specific load order which would not change often. If that becomes an issue we can move it into its own package.

(Speaking of non-version-specific, it might be nice to be able to define a non-version-specific MCConfiguration for this purpose, so that it's clear in the method that the version numbers aren't meant to be used. E.g. map _ MCConfiguration fromArray: #(repository ('http://source.squeakfoundation.org/39a') dependency ('Kernel') dependency ('Collections') ... etc. Although perhaps the initial versions or uuid's are needed for ancestry info? Anyway, not a big deal.)

- Doug

On Jul 10, 2005, at 5:40 AM, Bert Freudenberg wrote:

Am 10.07.2005 um 06:17 schrieb Doug Way:

...

This may become obvious when I look into it, but I assume this MC configuration will load the very latest versions of each package, not necessarily the specified versions?

No, installing always uses the version specified in the map:

MCConfiguration>>load loads exactly the versions specified in the configuration, overwriting even newer versions.

MCConfiguration>>merge merges exactly the versions specified in the configuration into the image.

MCConfiguration>>upgrade installs (*) exactly the versions specified in the configuration. However, it does not install a version if the one in the image is newer (that is, the version in the config map is an ancestor of the on in the image). (*) The preference #upgradeIsMerge governs the method for installing, that is, loading or merging. Normal users will want to use load, whereas system developers most surely do not want their local versions clobbered by an upgrade, so they'll enable that preference.

So if you look into a changeset installing a config map, it creates a config map and installs (using the #upgrade method) exactly those versions. You need that for various scenarios, just installing the latest packages when you are way behind would almost be guaranteed to not work.

...

(That's what I was looking for... a simple "load order" configuration and not a configuration that only loaded specific versions.) I think that's what the #upgrade does if I remember right, but I'll have to try it.

MCConfiguration>>updateFromRepositories modifies the configuration to point to the latest version it can find in the repositories. That is, it takes the old load order and plugs in the newest version of each package. It does not install any version.

So, to use an existing config map specifying the load order, but load the newest stuff, you'ld do:

map updateFromRepositories.
map upgrade.

(incidentally, that's what the code below says ;-)

...

I assume the CProjectBuilder class below is in a particular Tweak package. Meaning that if you wanted to change the list of packages in #defaultConfiguration, the #defaultConfiguration method in that package would be updated. (In other words, that particular package would sort of store the configuration.)

Yes, the class holding that should go into the "core" package or even into its own.

...

But I guess it wouldn't change that often, if it was simply used for the load order (and didn't have to be updated every time one package in the list bumped versions).

Yes, it changes rather rarely.

On Jul 17, 2005, at 4:43 PM, Bert Freudenberg wrote:

Am 17.07.2005 um 00:48 schrieb Doug Way:

...

2. The .mcd file does not exist. How is this intended to work?

A. Is the SqueakSource server supposed to generate the .mcd file on the fly, or maybe upon the saving of a new version? (Makes sense, that's a lot less bandwidth for these big packages.) B. Should the MCConfig code in the image not be trying to download .mcd files? C. Do we need to be saving .mcd files and not .mcz when we save new versions to the repository?

(I'm pretty sure the answer isn't C, that would be dumb. I'm guessing A.)

Yes it's A, a diff is generated on the fly when it is requested first, and then cached. Building a diff for version n against the previous n-1 versions would be O(n^2), and we might never need all. So it's only built on demand.

Makes sense.

So, the problem is that the squeaksource repository at sqf.org is not generating any .mcd files for whatever reason. It looks like this .mcd generation is a relatively new feature... is it something that was just added in the impara.de version of squeaksource? I see that http://squeaksource.com at berne does not seem to have the .mcd capability either, although at least it returns a "not found" error rather than hanging.

Actually, it's pretty easy to plug the URLs into a browser to test the various squeaksource repositories, since it's just a simple HTTP get. These links show how the various repositories behave:

works: http://source.impara.de/mc/Monticello-bf.260.mcz http://source.impara.de/mc/Monticello-bf.261.mcz http://source.impara.de/mc/Monticello-bf.261%28bf.260%29.mcd

works: http://source.squeakfoundation.org/39a/Kernel-dew.1.mcz http://source.squeakfoundation.org/39a/Kernel-md.5.mcz

doesn't work: (hangs!) http://source.squeakfoundation.org/39a/Kernel-md.5%28dew.1%29.mcd

works: http://squeaksource.com/Monticello/Monticello-avi.231.mcz http://squeaksource.com/Monticello/Monticello-avi.232.mcz

doesn't work: ("not found" error) http://squeaksource.com/Monticello/Monticello-avi.232%28avi.231%29.mcd

- Doug

On Mon, 18 Jul 2005 12:36:38 +0200, "Bert Freudenberg" bert@impara.de said:

Am 18.07.2005 um 06:00 schrieb Doug Way:

...

... works: http://source.impara.de/mc/Monticello-bf.260.mcz http://source.impara.de/mc/Monticello-bf.261.mcz http://source.impara.de/mc/Monticello-bf.261%28bf.260%29.mcd

works: http://source.squeakfoundation.org/39a/Kernel-dew.1.mcz http://source.squeakfoundation.org/39a/Kernel-md.5.mcz

doesn't work: (hangs!) http://source.squeakfoundation.org/39a/Kernel-md.5%28dew.1%29.mcd ...

Did you compare the package versions I sent around?

Actually, I didn't set up the SqueakSource installation at source.squeakfoundation.org... Cees & Avi did. So I'm not sure which versions are in that install. I'll bug them about it.

- Doug

On 7/23/05, Doug Way dway@mailcan.com wrote:

...

Actually, I didn't set up the SqueakSource installation at source.squeakfoundation.org... Cees & Avi did. So I'm not sure which versions are in that install. I'll bug them about it.

Hmm, haven't heard from them yet, they may be busy or on summer holiday.

Hey Doug,

Yeah, I unfortunately don't have a lot of bandwidth until I get back to Vancouver on Aug 16th. So if you want stuff done before then, it may be worthwhile for you to build or modify the SqueakSource install yourself. If there's something I can do to make access easier for you, though, let me know.

Avi

...

...

...

...

On Jul 4, 2005, at 1:09 AM, Doug Way wrote:

So, there is a new MonticelloConfigurations-dew.32 in the 39a repository at source.squeakfoundation.org

Hmm, actually, there are two ... if you look into the history of MonticelloConfigurations-dew.32.mcz, it has a version of the same name (but a different UUID) as its ancestor. That is going to be problematic as in squeaksource versions are not referenced by UUID but by name. So the second version overwrote the dictionary entry of the first. It might work if you upload the first version (with dew.31 as ancestor) again.

Btw, we copied your initial set of packages to our repository so we can start our 3.8 changes from there, which should make it easier to merge back later.

- Bert -

On Jul 12, 2005, at 11:46 PM, Doug Way wrote:

On Jul 12, 2005, at 5:57 AM, Bert Freudenberg wrote:

...

Hmm, actually, there are two ... if you look into the history of MonticelloConfigurations-dew.32.mcz, it has a version of the same name (but a different UUID) as its ancestor. That is going to be problematic as in squeaksource versions are not referenced by UUID but by name. So the second version overwrote the dictionary entry of the first. It might work if you upload the first version (with dew.31 as ancestor) again.

Yes, sorry about that. MonticelloConfigurations-bf.31 was the shared ancestor. Back when I made the change, I think I accidentally saved an initial MonticelloConfigurations-dew.32 without any changes, then I wanted to overwrite/replace that one with a new MonticelloConfigurations-dew.32 which actually did have the changes, so when it went to save MonticelloConfigurations-dew.33, I renamed it to .32 in the commit log dialog. I wondered at the time if this wasn't really proper. (Although it sounds like MC can handle it, but squeaksource can't.)

Ok, I went ahead and saved a new version MonticelloConfigurations-dew.33 in the 39a repository which should have a clean ancestry (going straight from .31). Try using that one if .32 gave you problems.

- Doug

Hi Bert. Ok, Cees set me up with access to the machine at http://source.squeakfoundation.org. Maybe I'll be able to fix this after all, possibly even before Avi gets back. ;)

Looking at the SqueakSource image that was running, it does have a few differences in the package list from yours. Here's the list:

DynamicBindings (DynamicBindings-gk.1) * KomHttpServer (KomHttpServer-gk.6) KomServices (KomServices-gk.2) Mewa (Mewa-al.13) * Monticello (Monticello-avi.182) SMBase (SMBase-gk.63) SMLoader (SMLoader-gk.24) Scheduler (Scheduler-jrp.18) Seaside2 (Seaside2-avi.86) * SqueakSource (SqueakSource-bf.146) TinyWiki (TinyWiki-lr.10)

Probably the Monticello & MonticelloConfiguration packages are the main issue. I'll try to track down the other few packages too, though.

I may see if I can get it working locally first, and then try to replace the one at sqf.org. (Hopefully the old accounts/data files would still be okay after replacing the squeaksource image. It looks like the .mcz files are all in separate dirs so those should be fine, which is the most important thing.)

- Doug

On Jul 12, 2005, at 5:42 AM, Bert Freudenberg wrote:

... So I just tried this. There must be something different on your squeaksource installation than on our's.

When I access "http://source.squeakfoundation.org/39a/MonticelloConfigurations- dew.31(bf.30).mcd" it just hangs. On our server, for "http://source.impara.de/iSqueak/MonticelloConfigurations- dew.31(bf.30).mcd'" I immediately get a 404 not found.

Here's the current list of my versions:

DynamicBindings (DynamicBindings-gk.1) KomHttpServer (KomHttpServer-gk.6) KomServices (KomServices-gk.2) Mewa (Mewa-al.13) Monticello (Monticello-bf.254) MonticelloConfigurations (MonticelloConfigurations-bf.26) PackageInfo-Base (PackageInfo-Base-bf.22) SMBase (SMBase-gk.63) SMLoader (SMLoader-gk.24) Seaside2 (Seaside2-avi.86) SqueakSource (SqueakSource-bf.146) SystemFixes (SystemFixes-bf.1) TinyWiki (TinyWiki-lr.10) TweakMC (TweakMC-bf.11)

• Bert -

Hello,

Right now I'm loading all the packages from the 39a repository into a test image, so I can put together an MCConfiguration.

I have a silly question, but why do we need MCConfiguration? Why not to have a package named SqueakPackages that will depends on all the package of the system. One will just have to load this package to install all the other.

Doug, how did you load "all the packages from the 39a repository" ? One by one ? Manually ?

Cheers, Alexandre

On 6/27/05, Avi Bryant avi.bryant@gmail.com wrote:

Ok, it's back up, and I'm slowly feeding it packages. For now you'll have to access it at port 9090 (http://source.squeakfoundation.org:9090/). Cees, when you get a chance, can you change the ProxyPass lines in the apache config to drop the /seaside/ss part?

Done.

Except that the major portion of work was the recategorization. (for anyone who missed the original thread, have a look at the wiki page on http://source.impara.de/iSqueak.html).

right. I guess you are referring to the Reorganize.3.8-6662.cs. There are just some categorizations. Bert, I did not deeply checked but I feel that actually it does not assign new category to classes/methods. For instance, the first line of the changeset is

SystemOrganization changeFromString: '(''Kernel-Chronology'' ChronologyConstants Date DateAndTime Duration Month Schedule Stopwatch Time TimeStamp Timespan TimeZone Week Year)

These classes are already included into the class category 'Kernel-Chronology'! Andreas worked on the system change notification, but I do not know what he did (I do not have time to dive into the iSqueak image). I suspect that it uses events triggered by the recategoryzation to create packages.

Alexandre

On Mon, 27 Jun 2005 18:26:21 +0200, "Bert Freudenberg" bert@impara.de said:

Am 27.06.2005 um 18:18 schrieb Alexandre Bergel:

...

right. I guess you are referring to the Reorganize.3.8-6662.cs. There are just some categorizations. Bert, I did not deeply checked but I feel that actually it does not assign new category to classes/ methods. For instance, the first line of the changeset is

SystemOrganization changeFromString: '(''Kernel-Chronology'' ChronologyConstants Date DateAndTime Duration Month Schedule Stopwatch Time TimeStamp Timespan TimeZone Week Year)

These classes are already included into the class category 'Kernel- Chronology'! Andreas worked on the system change notification, but I do not know what he did (I do not have time to dive into the iSqueak image). I suspect that it uses events triggered by the recategoryzation to create packages.

You need to inspect the change set more closely. It may well have redundant categorization of classes where nothing changed - this does not hurt anybody. But there are also many recategorizations of methods. I guess Andreas made the changeset by just filing out the categorization for everything in the system.

It is difficult to see exactly what changed in the recategorization, because the changeset includes the whole system as you say.

However, I did see that the Tests package was recategorized into KernelTests, CollectionTests, NetworkTests, etc., which is a significant change and seems like a good change.

Just curious, were any methods actually recategorized into mc-based extension method categories? (I didn't see any.) That would be a bigger change, as it would essentially move the method into a different package as an extension, even though it's still on the same class. A classic example would be moving String>>asUrl from the "converting" to the "*Network" method category, so that it ends up in the Network package. That's more of a real detangling/refactoring than a mere recategorization.

- Doug

Hi -

[Just subscribed to this list]

It is difficult to see exactly what changed in the recategorization, because the changeset includes the whole system as you say.

Here are all the changes:

a) Change the categories from "Foo-Bar-Tests" to "FooTests-Bar", e.g, "Kernel-Chronology-Tests" => "KernelTests-Chronology" "Collections-Streams-Tests" => "CollectionTests-Streams" The reason being that I want to be able to bootstrap a system without circular dependencies and moving the tests into their own categories makes it easy to load everything in the right order.

b) Merging of some remnant packages which need to be folded into Morphic and which never have been real standalone packages ("CustomEvents", "FlexibleVocabularies", ...)

c) Some arbitrary new top-level packages to get the size of the "System" package down to a reasonable level, including "Files", "Compiler", and "Compression"

d) Some reclassification to avoid remnants such as HTTPClient (a lone class) from "Framework-Download", FancyMailComposition from "EToy-Download" and then some.

The method reclassifications were all due b) e.g., removal of remnant packages.

Just curious, were any methods actually recategorized into mc-based extension method categories?

No. The idea was to get an *initial* structure going not to deal with fine-tuning package boundaries.

Cheers, - Andreas

I understand and I agree that having separated tests is good and avoid dependencies. Now I do not understand the naming conventions. I miss something but I do not know what. Why the tests of Kernel-Chronology are not simply KernelChronology-Tests? Because MC would put them under Kernel? but we could have a separate package for KernelChronologyTests?

Because creating fifty packages for each and every category ending with "-Test" wasn't TSTTCPW. The simplest thing was to reclassify the classes so we have a "Kernel" package and a "KernelTests" package, a "Collection" package and a "CollectionTests" package etc.

Again keep in mind that the goal of this exercise was to *fully* packagize the system - and because of this some compromises had to be made. If you want to make all of these separate packages, by all means, feel free to do so. But that was out of my scope.

Cheers, - Andreas

Realistically, it's going to take us a little while to get everything working smoothly with this new process.

It depends. I think it is important to have a clear how-to page that describe the process.

But we might as well dive in and get started. It's a big change, but I think it's flexible enough that we will be able to adjust the process if necessary, and we won't end up in a 3.3alpha modules situation.

Squeak evolved a lot since I started to use it. I do not thing that big changes are bad if they make life of people easier.

I will set up a new swiki page covering the new process, as a point of reference.

I am willing to help.

alexandre

I think it is a good start. I will definitely contribute to it once we have something that runs.

Alexandre

On Tue, Jun 28, 2005 at 02:02:28PM -0400, Doug Way wrote:

On Mon, 27 Jun 2005 18:22:11 +0200, "Alexandre Bergel" bergel@iam.unibe.ch said:

...

...

Realistically, it's going to take us a little while to get everything working smoothly with this new process.

It depends. I think it is important to have a clear how-to page that describe the process.

...

...

I will set up a new swiki page covering the new process, as a point of reference.

I am willing to help.

Thanks Alexandre. Ok, I've set up the swiki page for the new process:

http://minnow.cc.gatech.edu/squeak/5718

It's just an overview at this point, we can add some more specific how-to details soon. Also, it has the somewhat clumsy name "3.9 Packages+Update Stream Process", but I guess that's good enough for now.

• Doug

http://source.squeakfoundation.org:9090/ does not work, and http://source.squeakfoundation.org/ => "The proxy server received an invalid response from an upstream server."

Alexandre

On Tue, Jun 28, 2005 at 09:44:59AM +0200, stéphane ducasse wrote:

I cannot access

source.squeakfoundation.org may be this is my settings but I get The proxy server received an invalid response from an upstream server.

On 27 juin 05, at 17:59, Doug Way wrote:

...

On Mon, 27 Jun 2005 17:15:16 +0200, "Avi Bryant" avi.bryant@gmail.com said:

...

On 6/27/05, Alexandre Bergel bergel@iam.unibe.ch wrote:

...

But the Project 39a is still empty...

Ah, yes, it is. We do need to commit stuff there. I just didn't understand the purpose of a script that created new packages when we already have a changeset from Andreas that does so.

Right, we can just use Andreas' changeset to do this partitioning. (which is very similar to Alexandre's script anyway, I think)

...

I'll populate that project shortly.

Sounds good. By the way, does a SqueakSource "Developer" have access to do this initial populating (adding new packages), or do you need to be an Admin?

...

...

I have many changes related to the system change notification that are waiting...

Great.

Yes, that should give the new process a good workout. Hmm, how many packages do these change-notification changes cut across? :-)

Realistically, it's going to take us a little while to get everything working smoothly with this new process. But we might as well dive in and get started. It's a big change, but I think it's flexible enough that we will be able to adjust the process if necessary, and we won't end up in a 3.3alpha modules situation.

I will set up a new swiki page covering the new process, as a point of reference.

• Doug

Am 28.06.2005 um 10:36 schrieb Cees De Groot:

On 6/28/05, Alexandre Bergel bergel@iam.unibe.ch wrote:

...

http://source.squeakfoundation.org:9090/ does not work, and http://source.squeakfoundation.org/ => "The proxy server received an invalid response from an upstream server."

I've restarted the image.

Avi - do you want to investigate the crashes or shall I just hang the thing under daemontools' supervision so it automagically restarts?

What VM are you running? Anyway, I, too, run it in a loop, just in case (hadn't much time to investigate the crashes, yet):

#!/bin/sh DIR=`dirname $0` exec nohup sh -c "while true; do $DIR/runss ; sleep 5; done"

Also, if it hangs, you can just kill it. I renamed the vm to make it stand out in the process listing for that.

- Bert -

I know, I use them at home. But on this server I'm not the boss.

- Bert -

Am 28.06.2005 um 11:44 schrieb Cees De Groot:

I suggest you take a peek at daemontools. It does more-or-less the same, but much cleaner ;-)

The machine is running 3.7-7

On 6/28/05, Bert Freudenberg bert@impara.de wrote:

...

What VM are you running? Anyway, I, too, run it in a loop, just in case (hadn't much time to investigate the crashes, yet):

#!/bin/sh DIR=`dirname $0` exec nohup sh -c "while true; do $DIR/runss ; sleep 5; done"

Also, if it hangs, you can just kill it. I renamed the vm to make it stand out in the process listing for that.

• Bert -