Hi
today I realized (I was idiot or totally into my envy/store practices) that we cannot manage the image like we do for an application with MC (or we do not understand anything) or Envy/Store. Here are the notes I took when going step by step over a releasing a new image with marcus. I guess that it summarises somehow our questions and problems.
We (with marcus) are trying to understand how we can improve the way we harvest and produce new images with the new process. Now we are a bit stuck because I harvested too much. I always naively thought that we could take 3.8 and load only the latest version + some changes that where done via cs.
But this is not possible for all the stuff - (1) that can only be done via cs, - (2) that one package may rely on another one that may rely on the other. (here I want to understand how MC can help see below)
So if I understand correctly this means that to arrive to version x, we have to load all the configs (hence the MC deltas to speed up the process) until we load version x. I guess that this is the set up done at impara to manage iSqueak and Tweak. Bert is it correct?
Now this means that this is really tedious to have to - define - validate - a load order for all the packages - publish a new cs on the update stream - publish a new image
The implications with this setup is that we have to produce often an image (validate a configuration of changes working together) and this can be quite painful. At least they are a lot of try and error when there are a lot of dependencies between packages.
Bert what is the process that you have?
Do you have something like getMeTheLatestConfiguration that worked from MC (I guess that what you have is the cs script that are in the update stream and scripts a working configuration) Do you have automated scripts? We have problems to really understand the MCConfigurationBrowser Can we programmatically filled up the MCConfigurationBrowser?
For the second problem (2) above Avi do you know if creating a package named Squeak with all the subpackages as required would help minimizing the manual conflicts resolution. I thought that MC was supporting an atomic load so I would like to use that as much as possible.
Bert I have the impression that using package dependencies with a root package (ie havgin Squeak and all the image package as required) would help to minimize inter package dependencies, now why do you use your script (the explicit list of packages) instead of required package. Is it because you can push that in the update stream?
What I like with required packages is that we version a configuration and the atomic load (if I was not dreaming) Now with the scripts at the end of the process we have an image and in the cs the configurations. Am'I correct?
Stef and marcus
On Sep 18, 2005, at 8:01 AM, stéphane ducasse wrote:
For the second problem (2) above Avi do you know if creating a package named Squeak with all the subpackages as required would help minimizing the manual conflicts resolution. I thought that MC was supporting an atomic load so I would like to use that as much as possible.
Can you explain what you mean here by manual conflicts resolution?
I do think that using a Squeak package with dependencies, or alternatively just changing a Configuration to have a load strategy more like a package with dependencies, would be a good idea. When I recently tried updating a 3.9a image through a few configuration steps, it was bad enough that it took as long as it did, but it was awful that it stopped to ask me about loading over a modified package many times during the process - so I couldn't just walk away. This wouldn't happen if MC's multi-package-load mechanism were used instead of explicit ordering. It would also mean you wouldn't (usually) have to think about package ordering yourself. So it should speed up the process for everyone.
It also seems like configurations are maybe being posted to the update stream too frequently: I doubt I actually needed to go through so many intermediate steps to get to the final state. I assume this is why Impara uses the combination of explicitly posted configs, and just updating to the most recent versions in a given repository. Alternatively, when posting a new configuration to the stream, you should remove old ones that are no longer needed or replace them with no-ops.
This process is probably hardest on you guys that are doing the harvesting. From the point of view of both submitting patches to the /inbox and browsing through histories in a pre-packaged image, I very much like the new approach. So, from where I stand anyway, all this pain *is* worth it :)
Avi
On 18 sept. 05, at 19:38, Avi Bryant wrote:
On Sep 18, 2005, at 8:01 AM, stéphane ducasse wrote:
For the second problem (2) above Avi do you know if creating a package named Squeak with all the subpackages as required would help minimizing the manual conflicts resolution. I thought that MC was supporting an atomic load so I would like to use that as much as possible.
Can you explain what you mean here by manual conflicts resolution?
I mean try and error to find the order of the package. apparently, now I lost all the work I did on harvesting recently because I cannot load the morphic splitter works and merging I did.
I do think that using a Squeak package with dependencies, or alternatively just changing a Configuration to have a load strategy more like a package with dependencies, would be a good idea. When I recently tried updating a 3.9a image through a few configuration steps, it was bad enough that it took as long as it did, but it was awful that it stopped to ask me about loading over a modified package many times during the process - so I couldn't just walk away. This wouldn't happen if MC's multi-package-load mechanism were used instead of explicit ordering. It would also mean you wouldn't (usually) have to think about package ordering yourself. So it should speed up the process for everyone.
Exactly
It also seems like configurations are maybe being posted to the update stream too frequently: I doubt I actually needed to go through so many intermediate steps to get to the final state.
I do not know I follow what marcus did. And as soon as he fight with a configuration he created an image to save that state.
I assume this is why Impara uses the combination of explicitly posted configs, and just updating to the most recent versions in a given repository. Alternatively, when posting a new configuration to the stream, you should remove old ones that are no longer needed or replace them with no-ops.
I think that we simply need HELP here because I do not know all the subtleties and managing the image is not just a simple application. For example, just loading the latest does not work with the latest image with publish today and the harvesting I did since last week. Just trying to load Morphic breakkkkkkks.
This process is probably hardest on you guys that are doing the harvesting.
Yes so this means that we will not be able to do it on the long run. because we should check too much stuff, the quality, the fixes, ... and find a correct loading order.
From the point of view of both submitting patches to the /inbox and browsing through histories in a pre-packaged image, I very much like the new approach. So, from where I stand anyway, all this pain *is* worth it :)
Yes but as I said if harvesting is harder then you can fill up the inbox, nobody will look at it. and especially if this is difficult. So what are the concrete suggestion to remove as much as pain for the ordering as possible. So I will try to create a Squeak package and try that ... but again I think that we need help.
Stef
Avi Bryant wrote:
I do think that using a Squeak package with dependencies, or alternatively just changing a Configuration to have a load strategy more like a package with dependencies, would be a good idea. When I recently tried updating a 3.9a image through a few configuration steps, it was bad enough that it took as long as it did, but it was awful that it stopped to ask me about loading over a modified package many times during the process - so I couldn't just walk away. This wouldn't happen if MC's multi-package-load mechanism were used instead of explicit ordering. It would also mean you wouldn't (usually) have to think about package ordering yourself. So it should speed up the process for everyone.
Please say more about this, since it's been a while that I have used dependencies in MC at all. About load strategies: I don't know what strategy MC uses but unless it gets this right every single time (which I doubt) having configurations is vastly advantageous. Usually, I have a "master configuration" (for a project like Tweak) which lists the fifty or so packages in what I consider "load order", e.g., a unique order for loading the packages that includes eventual dependencies. If I post a new configuration I simply "update" the old oonfiguration from either image or packages and post it. It's a very simple process but if there's any need I can re-order the packages for that specific post.
About it being "bad enough": I agree. I think the current model is not very well suited for the current set of requirements. For example, posting confs explicitly can be a good idea if you're doing active development in a repository and only want people to see certain changes but if you don't, there is really no point in it. One might as well just have people update the packages implicitly whenever possible (e.g., no load conflicts).
About being asked for loading over a modified package: That seems easy to fix if it were such a big issue but I still don't see if you can walk away if you have a modified package. You would at least have to have a look at the conflicts, don't you?
It also seems like configurations are maybe being posted to the update stream too frequently: I doubt I actually needed to go through so many intermediate steps to get to the final state. I assume this is why Impara uses the combination of explicitly posted configs, and just updating to the most recent versions in a given repository.
That is correct.
Cheers, - Andreas
On Sep 18, 2005, at 12:53 PM, Andreas Raab wrote:
I do think that using a Squeak package with dependencies, or alternatively just changing a Configuration to have a load strategy more like a package with dependencies, would be a good idea. When I recently tried updating a 3.9a image through a few configuration steps, it was bad enough that it took as long as it did, but it was awful that it stopped to ask me about loading over a modified package many times during the process - so I couldn't just walk away. This wouldn't happen if MC's multi- package-load mechanism were used instead of explicit ordering. It would also mean you wouldn't (usually) have to think about package ordering yourself. So it should speed up the process for everyone.
Please say more about this, since it's been a while that I have used dependencies in MC at all. About load strategies: I don't know what strategy MC uses but unless it gets this right every single time (which I doubt) having configurations is vastly advantageous. Usually, I have a "master configuration" (for a project like Tweak) which lists the fifty or so packages in what I consider "load order", e.g., a unique order for loading the packages that includes eventual dependencies. If I post a new configuration I simply "update" the old oonfiguration from either image or packages and post it. It's a very simple process but if there's any need I can re-order the packages for that specific post.
Let's say I have two packages, A and B. I have a new configuration in which these two packages have exchanged methods: one method from A has moved to B, and one method from B has moved to A. I try to load that configuration. Either load order is equivalent, but let's say that we've specified that the order is A, B.
What happens is this: first A is loaded, taking one method from B and losing one method itself. Because it took a method from B, B is now marked as being modified. Next, B is loaded, but it has a modified flag, and so the user is asked whether it's ok to ignore those modifications. This is confusing, since the user never has modified B. It's also irritating because the load can't proceed unsupervised.
Ok, now a more complicated scenario: A has added a Foo class and a BarSub class. B has added a FooSub class and a Bar class. FooSub is a subclass of Foo, and BarSub is a subclass of Bar. This is admittedly contrived, and arguably bad design, but there's no sequential load order that will work for these two packages.
Both of these cases are solved by simply asking MC to load or merge the two packages together, rather than asking it to load them sequentially. The load strategy it uses is very simple: a superclass always needs to be loaded before its subclasses, and a class always needs to be loaded before its methods. Additions are always done before removals, and removals are done with the inverse rules (first remove methods, then classes, then superclasses).
What you don't get to do in that case is specify anything about the load order yourself. I think the cases where specifying this is necessary are going to be very rare, but certainly they will happen. That's what the update stream is useful for: simply put two configurations in (one of them, maybe, with only a single package) to ensure that loads happen in the correct sequence. This should only be needed for delicate migrations, I don't think it should ever be needed for every time a group of packages is loaded together.
Avi
Avi Bryant wrote:
Let's say I have two packages, A and B. I have a new configuration in which these two packages have exchanged methods: one method from A has moved to B, and one method from B has moved to A. I try to load that configuration. Either load order is equivalent, but let's say that we've specified that the order is A, B.
Ah, yes. *That* problem ;-) I've seen it often enough, usually in the form of classes being moved between packages. And it's a pain to deal with, I agree.
Both of these cases are solved by simply asking MC to load or merge the two packages together, rather than asking it to load them sequentially. The load strategy it uses is very simple: a superclass always needs to be loaded before its subclasses, and a class always needs to be loaded before its methods. Additions are always done before removals, and removals are done with the inverse rules (first remove methods, then classes, then superclasses).
Ah, interesting. I had no idea. But yes, that makes a lot of sense.
What you don't get to do in that case is specify anything about the load order yourself. I think the cases where specifying this is necessary are going to be very rare, but certainly they will happen.
I agree. I think the ratio for configs per package versions is somewhere between 1:10 and 1:20 in Tweak.
Cheers, - Andreas
On Sep 18, 2005, at 4:07 PM, Avi Bryant wrote:
Both of these cases are solved by simply asking MC to load or merge the two packages together, rather than asking it to load them sequentially. The load strategy it uses is very simple: a superclass always needs to be loaded before its subclasses, and a class always needs to be loaded before its methods. Additions are always done before removals, and removals are done with the inverse rules (first remove methods, then classes, then superclasses).
What you don't get to do in that case is specify anything about the load order yourself. I think the cases where specifying this is necessary are going to be very rare, but certainly they will happen. That's what the update stream is useful for: simply put two configurations in (one of them, maybe, with only a single package) to ensure that loads happen in the correct sequence. This should only be needed for delicate migrations, I don't think it should ever be needed for every time a group of packages is loaded together.
One thing that isn't addressed here is initialization order. If package A depends on package B, A should be completely initialized before initialization of B begins. I don't think we do this correctly at the moment.
Otherwise, I completely agree. We should use sequences of Configurations in the update stream for those special cases where we need specific intermediate states. In all other cases, MC's internal load ordering should be fine.
Colin
Hi colin
I just discussed this point with avi. My take on that is a bit extreme (and does not fit a system like squeak) but I would say that per design you should not have initialization order.
Both of these cases are solved by simply asking MC to load or merge the two packages together, rather than asking it to load them sequentially. The load strategy it uses is very simple: a superclass always needs to be loaded before its subclasses, and a class always needs to be loaded before its methods. Additions are always done before removals, and removals are done with the inverse rules (first remove methods, then classes, then superclasses).
What you don't get to do in that case is specify anything about the load order yourself. I think the cases where specifying this is necessary are going to be very rare, but certainly they will happen. That's what the update stream is useful for: simply put two configurations in (one of them, maybe, with only a single package) to ensure that loads happen in the correct sequence. This should only be needed for delicate migrations, I don't think it should ever be needed for every time a group of packages is loaded together.
One thing that isn't addressed here is initialization order. If package A depends on package B, A should be completely initialized before initialization of B begins. I don't think we do this correctly at the moment.
Otherwise, I completely agree. We should use sequences of Configurations in the update stream for those special cases where we need specific intermediate states. In all other cases, MC's internal load ordering should be fine.
Colin
stéphane ducasse wrote:
So if I understand correctly this means that to arrive to version x, we have to load all the configs (hence the MC deltas to speed up the process) until we load version x.
This is important in order to be able to do changes that require specific intermediate steps, exactly like what is planned for the entry of Traits into 3.9a.
I guess that this is the set up done at impara to manage iSqueak and Tweak. Bert is it correct?
Now this means that this is really tedious to have to - define - validate - a load order for all the packages - publish a new cs on the update stream - publish a new image
The implications with this setup is that we have to produce often an image (validate a configuration of changes working together) and this can be quite painful. At least they are a lot of try and error when there are a lot of dependencies between packages.
About creating the cs and new image - this should probably be automated. Something like "write cs describing current image, get the last published image, load updates, save new image to web space". I don't understand why the load order is necessary and what trial and error comes out of that.
But maybe Avi's response solves that part of the problem.
A couple of notes and proposals of my own - 1. I think one way to make you lives easier is to try to use only MC as much as possible. Tell people you accept only mcz/mcd files, for the existing packages. This will take time, but it will mean that people will think about their changes and what they affect, and its one less thing for you to figure out. If you get a CS, do you go over it manually and check that methods are marked as extensions where needed? if you require package versions, people are likelier to think of that themselves. 2. Use proper repositories, not Mantis. Mantis can contain a link to an mcz file in a repository, that is as easy for you to use as an attachment. But it helps accountability, because then anyone can get the original submitted version and inspect it. So tell people to put their versions up on source.squeakfoundation.org/inbox or to create themselves an account a personal project there, and then put up all their proposed changes as mcz files. 3. Tell everyone that care about maintaining the image to register for the sources.squeakfoundation.org RSS feed (I just found out about this). That way they get notified for every version that gets put out. That way people see what changes are proposed, and there's a better chance that good changes will be endorsed/extended, and bad changes will be seen and commented on. This is now the equivalent of reading the BFAV, except you can do it from Thunderbird.
This policy direction of moving towards using MC and SqS more exclusively for the management of the image are good because: 1. The current hybrid policy is too complex (as seen for example by the confusion caused by looking in changesets for traceability) 2. We can improve the MC tools to improve the image maintenance process.
For example, we can make one repository inspector that shows quickly all the new versions of everything that people suggest, and then harvesting will become easier in that way too.
Daniel
I like your suggestions definitively the way to go.
Stef
On 18 sept. 05, at 20:57, Daniel Vainsencher wrote:
stéphane ducasse wrote:
So if I understand correctly this means that to arrive to version x, we have to load all the configs (hence the MC deltas to speed up the process) until we load version x.
This is important in order to be able to do changes that require specific intermediate steps, exactly like what is planned for the entry of Traits into 3.9a.
I guess that this is the set up done at impara to manage iSqueak and Tweak. Bert is it correct? Now this means that this is really tedious to have to - define - validate - a load order for all the packages - publish a new cs on the update stream - publish a new image The implications with this setup is that we have to produce often an image (validate a configuration of changes working together) and this can be quite painful. At least they are a lot of try and error when there are a lot of dependencies between packages.
About creating the cs and new image - this should probably be automated. Something like "write cs describing current image, get the last published image, load updates, save new image to web space". I don't understand why the load order is necessary and what trial and error comes out of that.
But maybe Avi's response solves that part of the problem.
A couple of notes and proposals of my own -
- I think one way to make you lives easier is to try to use only
MC as much as possible. Tell people you accept only mcz/mcd files, for the existing packages. This will take time, but it will mean that people will think about their changes and what they affect, and its one less thing for you to figure out. If you get a CS, do you go over it manually and check that methods are marked as extensions where needed? if you require package versions, people are likelier to think of that themselves. 2. Use proper repositories, not Mantis. Mantis can contain a link to an mcz file in a repository, that is as easy for you to use as an attachment. But it helps accountability, because then anyone can get the original submitted version and inspect it. So tell people to put their versions up on source.squeakfoundation.org/inbox or to create themselves an account a personal project there, and then put up all their proposed changes as mcz files. 3. Tell everyone that care about maintaining the image to register for the sources.squeakfoundation.org RSS feed (I just found out about this). That way they get notified for every version that gets put out. That way people see what changes are proposed, and there's a better chance that good changes will be endorsed/extended, and bad changes will be seen and commented on. This is now the equivalent of reading the BFAV, except you can do it from Thunderbird.
This policy direction of moving towards using MC and SqS more exclusively for the management of the image are good because:
- The current hybrid policy is too complex (as seen for example by
the confusion caused by looking in changesets for traceability) 2. We can improve the MC tools to improve the image maintenance process.
For example, we can make one repository inspector that shows quickly all the new versions of everything that people suggest, and then harvesting will become easier in that way too.
Daniel
On Sep 18, 2005, at 11:57 AM, Daniel Vainsencher wrote:
- Tell everyone that care about maintaining the image to register
for the sources.squeakfoundation.org RSS feed (I just found out about this). That way they get notified for every version that gets put out. That way people see what changes are proposed, and there's a better chance that good changes will be endorsed/extended, and bad changes will be seen and commented on. This is now the equivalent of reading the BFAV, except you can do it from Thunderbird.
Good point. It would be nice if this feed showed the full package summary (with ancestors etc) rather than just the commit log. Might even be good to show textual diffs in it.
Avi
Hi -
The implications with this setup is that we have to produce often an image (validate a configuration of changes working together) and this can be quite painful. At least they are a lot of try and error when there are a lot of dependencies between packages.
In my experience that's a sign that a) you haven't found a good package structure and/or b) you don't test early enough. In the first case the effect is that higher-level changes get intermangled with lower-level changes (that do in fact require careful folding into a running system). An indication towards that point is the effect that the Kernel package has (I think) the most revisions, where clearly it is one of the most stable packages (if it isn't we're in big trouble) and should have the least number of modifications. The same is probably true for Morphic at this point - you *need* a better package structure for Morphic which makes clear when you touch the critical parts (requiring very careful testing) and when the non-critical ones. In comparison, in Tweak I can typically tell by just looking at the package name whether this will require any effort to make sure it loads correctly or not.
It is also really important to test "early enough" whether you can load a package or not after you've done a modification. Unfortunately, you guys are somewhat handicapped since you don't have an automatic default update mechanism like we're using in Tweak. When I post a new version of a package that has even the slightest chance to affect anything I immediately get a fresh image, hit update and test it. Cost me perhaps twenty seconds but if it doesn't work I can easily either post an appropriate configuration or make a change to the package to make it load and post a new version of the package. I really couldn't imagine to work like you do, e.g., make a whole set of new package versions, try to load and figure out where and why it dies.
Bert I have the impression that using package dependencies with a root package (ie havgin Squeak and all the image package as required) would help to minimize inter package dependencies, now why do you use your script (the explicit list of packages) instead of required package. Is it because you can push that in the update stream?
Early on we looked at using dependencies but they didn't work very well for what we wanted to do. Most importantly, if you have a newer version of a package it will load the older version instead (VERY bad). There were also some serious issues with nested dependencies (Bert and I had a lot of fun figuring that out). And keep in mind that dependencies are not as good as a load-order since they typically only specify a partial relation (A dependsOn: X and B dependsOn: X does not specify whether A is loaded before B or vice versa) whereas we wanted to make sure that the updating process uses a single well-defined load order for everyone.
Cheers, - Andreas
So this is what I'm working on - MCConfigurations, by using merge instead of load when it is needed, solve the "older dependency problem" (at least in the context of developers).
However to me the fact that they specify a load order, instead of loading all those versions simultaneously, looks like a bug. The way I've been thinking about ordering issues is that if you need to specify an order because of reference dependencies, that's what simultaneous loading is for. If you need to specify an order for bootstrapping reasons, then you should have a sequence of configurations that each specify a step that works from the previous step, and allow those steps to define the necessary orderings of changes.
So unless there's another reason for the total order, or a problem with one of the above, it seems we can avoid specifying and maintaining a load order, and also be able to load circularly dependent packages.
Of course circularly dependent packages should be eventually dealt with, but that should not determine our upgrade mechanism, especially when MC already handles simu-load.
So I'm proposing to have MCConfigs to use simu-load for users and simu-merge developers, and ignore any specified order. Avi and have sketched the needed changes out, I'll work on it soon.
Daniel
Andreas Raab wrote:
Early on we looked at using dependencies but they didn't work very well for what we wanted to do. Most importantly, if you have a newer version of a package it will load the older version instead (VERY bad). There were also some serious issues with nested dependencies (Bert and I had a lot of fun figuring that out). And keep in mind that dependencies are not as good as a load-order since they typically only specify a partial relation (A dependsOn: X and B dependsOn: X does not specify whether A is loaded before B or vice versa) whereas we wanted to make sure that the updating process uses a single well-defined load order for everyone.
Cheers,
- Andreas
Hi -
Daniel Vainsencher wrote:
However to me the fact that they specify a load order, instead of loading all those versions simultaneously, looks like a bug. The way I've been thinking about ordering issues is that if you need to specify an order because of reference dependencies, that's what simultaneous loading is for. If you need to specify an order for bootstrapping reasons, then you should have a sequence of configurations that each specify a step that works from the previous step, and allow those steps to define the necessary orderings of changes.
Yes, you can do that. But what that means is more work since many implicit updates that work because of the implicit order will no longer work. This would be a net loss. Also, I do not understand what you guys refer to when you talk about "simultaneous" loading. What exactly does that mean? One of the advantages of packages is that you have control over granularity and that extends to loading them - the fact that you *can* specify the order in which packages get loaded is an advantage in my eyes (I certainly have used it often enough).
So unless there's another reason for the total order, or a problem with one of the above, it seems we can avoid specifying and maintaining a load order, and also be able to load circularly dependent packages.
The need to "specify and maintain a load order" seems strikingly similar to "specify and maintain dependencies" ;-) But in any case, this is all about the cost-benefit ratio. As long as dependencies don't get into the way (so far they have) and offer actual advantages (so far they haven't) I'm all ears. Let's just not be too quick throwing the configurations out with the bath water - I'm not convinced yet that dependencies will offer any benefits over configurations; simply because I think having a well-defined (and accessible) load-order is vastly advantaguous to trying to figure out how to load what and deal with the effects when things go wrong. But we'll see and I'm certainly willing to give it a shot.
So I'm proposing to have MCConfigs to use simu-load for users and simu-merge developers, and ignore any specified order. Avi and have sketched the needed changes out, I'll work on it soon.
Again, please explain "simu-load" and "simu-merge". This discussion is the first I ever heard about it and I just don't know what this means exactly.
But to summarize: Am I correct in understanding that you are proposing to change loading configs in order to use dependencies to create configurations? If not, I fail to see how this last paragraph relates to dependencies ;-)
Cheers, - Andreas
I'm creating an MCConfiguration2 class, so we can play with them in parallel.
Yes, you can do that. But what that means is more work since many implicit updates that work because of the implicit order will no longer work.
To my current understanding the only implicit updates that will affected are ones where having distinct configurations would probably be better documentation of what is going on anyway, and that might or might not turn out to be less than you think.
This would be a net loss.
I don't know yet.
Daniel
Andreas Raab wrote:
Hi -
Daniel Vainsencher wrote:
However to me the fact that they specify a load order, instead of loading all those versions simultaneously, looks like a bug. The way I've been thinking about ordering issues is that if you need to specify an order because of reference dependencies, that's what simultaneous loading is for. If you need to specify an order for bootstrapping reasons, then you should have a sequence of configurations that each specify a step that works from the previous step, and allow those steps to define the necessary orderings of changes.
Also, I do not understand what you guys
refer to when you talk about "simultaneous" loading. What exactly does that mean? One of the advantages of packages is that you have control over granularity and that extends to loading them - the fact that you *can* specify the order in which packages get loaded is an advantage in my eyes (I certainly have used it often enough).
So unless there's another reason for the total order, or a problem with one of the above, it seems we can avoid specifying and maintaining a load order, and also be able to load circularly dependent packages.
The need to "specify and maintain a load order" seems strikingly similar to "specify and maintain dependencies" ;-) But in any case, this is all about the cost-benefit ratio. As long as dependencies don't get into the way (so far they have) and offer actual advantages (so far they haven't) I'm all ears. Let's just not be too quick throwing the configurations out with the bath water - I'm not convinced yet that dependencies will offer any benefits over configurations; simply because I think having a well-defined (and accessible) load-order is vastly advantaguous to trying to figure out how to load what and deal with the effects when things go wrong. But we'll see and I'm certainly willing to give it a shot.
So I'm proposing to have MCConfigs to use simu-load for users and simu-merge developers, and ignore any specified order. Avi and have sketched the needed changes out, I'll work on it soon.
Again, please explain "simu-load" and "simu-merge". This discussion is the first I ever heard about it and I just don't know what this means exactly.
But to summarize: Am I correct in understanding that you are proposing to change loading configs in order to use dependencies to create configurations? If not, I fail to see how this last paragraph relates to dependencies ;-)
Cheers,
- Andreas
On Sep 18, 2005, at 1:29 PM, Andreas Raab wrote:
So I'm proposing to have MCConfigs to use simu-load for users and simu-merge developers, and ignore any specified order. Avi and have sketched the needed changes out, I'll work on it soon.
Again, please explain "simu-load" and "simu-merge". This discussion is the first I ever heard about it and I just don't know what this means exactly.
But to summarize: Am I correct in understanding that you are proposing to change loading configs in order to use dependencies to create configurations? If not, I fail to see how this last paragraph relates to dependencies ;-)
I think dependencies are actually a red herring: I think we all agree that they're broken and that external configs are a better way to go. So let me focus on the "simu-load/merge" stuff.
At its lower levels - the parts that actually load code into the image - Monticello doesn't know anything about packages. All it knows is, basically, "there's this set of definitions in the image, and this other set of definitions coming from the outside, and I need to adjust the image's set so that it matches the external set". Merge works the same way "I need to merge the image's set with this other set".
Now, the usual case is that the set of definitions in the image corresponds to all of the methods currently in the image in a single package, and the set of definitions coming in from the outside corresponds to those in a single package version file. But there's actually no reason that has to be so: you can throw as many definitions into that pool at once as you want. So the set from the image could be the union of the definitions from several packages, as could the set coming in from the outside.
Within that pool of definitions, package boundaries are unknown and ignored. The load order is determined for the pool as a whole, using the rules I mentioned earlier. So if you throw a whole bunch of packages into the pool, it will figure out a load ordering at method/ class granularity, rather than at package granularity.
Concretely, if you look at MCVersion>>load, it boils down to this:
MCVersion>>load MCVersionLoader new addVersion: aVersion; load
If you want to load more than one version at once, you simply need to do this:
MCVersionLoader new addVersion: versionA; addVersion: versionB; load
MCVersionMerger works similarly.
What I'm proposing and Daniel is working on is that we modify MCConfiguration to use this mechanism rather than an explicit load order.
Avi
Avi Bryant wrote: [... snip ...]
Within that pool of definitions, package boundaries are unknown and ignored. The load order is determined for the pool as a whole, using the rules I mentioned earlier. So if you throw a whole bunch of packages into the pool, it will figure out a load ordering at method/ class granularity, rather than at package granularity.
Thanks. That explains it, and I can see why this would be useful in particular in a situation where things get moved between various packages.
What I'm proposing and Daniel is working on is that we modify MCConfiguration to use this mechanism rather than an explicit load order.
How hard would it be to have this governed by a preference to begin with? I would like to find out whether a) this solves the problems Stef was seeing (the way I understand it they should) and b) what effect it would have if I just do a full update of Tweak using that mechanism (e.g., try to find out if would break on any of the updates we've put out so far).
Cheers, - Andreas
Hi guys
as a case study do the following: take the latest image we put on the ftp then
- create squeak package and add all the packages as required - save squeak.1 package - take the cs from http://webs.sinectis.com.ar/jmvuletich/ MorphicSplitters01.st - load it add the new package to squeak - save squeak.2 package - quit - take a fresh image and try to load. with me it failed. May be I should remove the dependencies between the package that the morphic splitters defined. I will redo the experience to check.
Now I tried to find a load order for the packages and I failed too. So now I'm stuck and I guess that for now. I consider MorphicSplitters not usable
Read more: Re: [ANN] [RFC] MorphicSplitters phase 1 completed
Stef
Ok, so we've done this exercise with simul-merge, also encountered the same kinds of problems. Now it looks like its working - we got an image with the MorphSplit work merged in.
You can try it out with MC version 274 from squeaksource. at this point use merge not load, because merge is what we looked at.
Avi and Daniel
stéphane ducasse wrote:
Hi guys
as a case study do the following: take the latest image we put on the ftp then
- create squeak package and add all the packages as required - save squeak.1 package - take the cs from http://webs.sinectis.com.ar/jmvuletich/MorphicSplitters01.st - load it add the new package to squeak - save squeak.2 package - quit - take a fresh image and try to load. with me it failed. May be I should remove the dependencies between the package that the morphic splitters defined. I will redo the experience to check.
Now I tried to find a load order for the packages and I failed too. So now I'm stuck and I guess that for now. I consider MorphicSplitters not usable
Read more: Re: [ANN] [RFC] MorphicSplitters phase 1 completed
Stef
This is an excellent news!
On 19 sept. 05, at 12:40, Daniel Vainsencher wrote:
Ok, so we've done this exercise with simul-merge, also encountered the same kinds of problems. Now it looks like its working - we got an image with the MorphSplit work merged in.
You can try it out with MC version 274 from squeaksource. at this point use merge not load, because merge is what we looked at.
Avi and Daniel
stéphane ducasse wrote:
Hi guys as a case study do the following: take the latest image we put on the ftp then - create squeak package and add all the packages as required - save squeak.1 package - take the cs from http://webs.sinectis.com.ar/jmvuletich/ MorphicSplitters01.st - load it add the new package to squeak - save squeak.2 package - quit - take a fresh image and try to load. with me it failed. May be I should remove the dependencies between the package that the morphic splitters defined. I will redo the experience to check. Now I tried to find a load order for the packages and I failed too. So now I'm stuck and I guess that for now. I consider MorphicSplitters not usable Read more: Re: [ANN] [RFC] MorphicSplitters phase 1 completed Stef
Yes, small relevant detail. The way we did it is | vm names | names _ 'EToys-dvf.1.mcz Kernel-dvf.35.mcz Network-dvf.16.mcz Tools-dvf.28.mcz MorphicExtras-dvf.1.mcz Movies-dvf.2.mcz FlexibleVocabularies-dvf.1.mcz MorphicTests-dvf.4.mcz BalloonMMFlash-dvf.1.mcz Nebraska-dvf.3.mcz ST80-dvf.17.mcz StarSqueak-dvf.3.mcz Graphics-dvf.13.mcz Files-dvf.8.mcz Speech-dvf.4.mcz System-dvf.41.mcz Balloon-dvf.5.mcz Multilingual-dvf.6.mcz Sound-dvf.2.mcz SMBase-dvf.69.mcz Compiler-dvf.12.mcz Protocols-dvf.4.mcz Collections-dvf.29.mcz Morphic-dvf.45.mcz' findTokens: ' ', String cr.
vm _ MCVersionMerger new. names do: [:fn | vm addVersion: (self loadVersionFromFileNamed: fn)]. vm merge
Where the dvf versions are the post morphic-split versions. The snippet above uses the same simultaneous merging that merging Squeak with lots of dependencies should use. That's pretty much what's going to be in the modified configs, except now its time for me to go catch a bus.
Daniel
stéphane ducasse wrote:
This is an excellent news!
On 19 sept. 05, at 12:40, Daniel Vainsencher wrote:
Ok, so we've done this exercise with simul-merge, also encountered the same kinds of problems. Now it looks like its working - we got an image with the MorphSplit work merged in.
You can try it out with MC version 274 from squeaksource. at this point use merge not load, because merge is what we looked at.
Avi and Daniel
stéphane ducasse wrote:
Hi guys as a case study do the following: take the latest image we put on the ftp then - create squeak package and add all the packages as required - save squeak.1 package - take the cs from http://webs.sinectis.com.ar/jmvuletich/ MorphicSplitters01.st - load it add the new package to squeak - save squeak.2 package - quit - take a fresh image and try to load. with me it failed. May be I should remove the dependencies between the package that the morphic splitters defined. I will redo the experience to check. Now I tried to find a load order for the packages and I failed too. So now I'm stuck and I guess that for now. I consider MorphicSplitters not usable Read more: Re: [ANN] [RFC] MorphicSplitters phase 1 completed Stef
Hi
yesterday I loaded the new version of MC and selected my old squeak. 2.mcz that was not loading previously and I did merge....
It works until I got an error like cannot find files....and I could not understand why. So I wanted to redo the complete experiement today but kilana was out of reach. I will retry locally and if it works I will publish that on 39a
should I use your script instead of clicking on the merge button? Setf
On 19 sept. 05, at 16:33, Daniel Vainsencher wrote:
Yes, small relevant detail. The way we did it is | vm names | names _ 'EToys-dvf.1.mcz Kernel-dvf.35.mcz Network-dvf.16.mcz Tools-dvf.28.mcz MorphicExtras-dvf.1.mcz Movies-dvf.2.mcz FlexibleVocabularies-dvf.1.mcz MorphicTests-dvf.4.mcz BalloonMMFlash-dvf.1.mcz Nebraska-dvf.3.mcz ST80-dvf.17.mcz StarSqueak-dvf.3.mcz Graphics-dvf.13.mcz Files-dvf.8.mcz Speech-dvf.4.mcz System-dvf.41.mcz Balloon-dvf.5.mcz Multilingual-dvf.6.mcz Sound-dvf.2.mcz SMBase-dvf.69.mcz Compiler-dvf.12.mcz Protocols-dvf.4.mcz Collections-dvf.29.mcz Morphic-dvf.45.mcz' findTokens: ' ', String cr.
vm _ MCVersionMerger new. names do: [:fn | vm addVersion: (self loadVersionFromFileNamed: fn)]. vm merge
Where the dvf versions are the post morphic-split versions. The snippet above uses the same simultaneous merging that merging Squeak with lots of dependencies should use. That's pretty much what's going to be in the modified configs, except now its time for me to go catch a bus.
Daniel
stéphane ducasse wrote:
This is an excellent news! On 19 sept. 05, at 12:40, Daniel Vainsencher wrote:
Ok, so we've done this exercise with simul-merge, also encountered the same kinds of problems. Now it looks like its working - we got an image with the MorphSplit work merged in.
You can try it out with MC version 274 from squeaksource. at this point use merge not load, because merge is what we looked at.
Avi and Daniel
stéphane ducasse wrote:
Hi guys as a case study do the following: take the latest image we put on the ftp then - create squeak package and add all the packages as required - save squeak.1 package - take the cs from http://webs.sinectis.com.ar/jmvuletich/ MorphicSplitters01.st - load it add the new package to squeak - save squeak.2 package - quit - take a fresh image and try to load. with me it failed. May be I should remove the dependencies between the package that the morphic splitters defined. I will redo the experience to check. Now I tried to find a load order for the packages and I failed too. So now I'm stuck and I guess that for now. I consider MorphicSplitters not usable Read more: Re: [ANN] [RFC] MorphicSplitters phase 1 completed Stef
On Sep 20, 2005, at 10:53 PM, stéphane ducasse wrote:
Hi
yesterday I loaded the new version of MC and selected my old squeak. 2.mcz that was not loading previously and I did merge....
It works until I got an error like cannot find files....and I could not understand why. So I wanted to redo the complete experiement today but kilana was out of reach. I will retry locally and if it works I will publish that on 39a
should I use your script instead of clicking on the merge button? Setf
The merge button should work - but it sounds like there was some problem with your repository configuration. Simplest would be to make sure that all the relevant .mcz files are in your package-cache directory before you start the merge.
Avi
I had a bit of that when I was doing it, in my case because some packages were needed from source.sqf.org which was inaccessible at the time.
Discover what packages/versions it wants, and make sure you have repositories with those in the relevant repository group (this was the quick solution for me, switched to source2), or do like Avi says and get everything before hand.
Daniel
Avi Bryant wrote:
On Sep 20, 2005, at 10:53 PM, stéphane ducasse wrote:
Hi
yesterday I loaded the new version of MC and selected my old squeak. 2.mcz that was not loading previously and I did merge....
It works until I got an error like cannot find files....and I could not understand why. So I wanted to redo the complete experiement today but kilana was out of reach. I will retry locally and if it works I will publish that on 39a
should I use your script instead of clicking on the merge button? Setf
The merge button should work - but it sounds like there was some problem with your repository configuration. Simplest would be to make sure that all the relevant .mcz files are in your package-cache directory before you start the merge.
Avi
Now I did the following:
Loaded morphic splitters added all the packages as required packages of Sqeuak and saved
then took back a new image, loaded MC274 and loaded the squeak.mcz file obtained previously and it froze squeak.
I'm redoing the experiment
Stef
On 21 sept. 05, at 08:20, Daniel Vainsencher wrote:
I had a bit of that when I was doing it, in my case because some packages were needed from source.sqf.org which was inaccessible at the time.
Discover what packages/versions it wants, and make sure you have repositories with those in the relevant repository group (this was the quick solution for me, switched to source2), or do like Avi says and get everything before hand.
Daniel
Avi Bryant wrote:
On Sep 20, 2005, at 10:53 PM, stéphane ducasse wrote:
Hi
yesterday I loaded the new version of MC and selected my old squeak. 2.mcz that was not loading previously and I did merge....
It works until I got an error like cannot find files....and I could not understand why. So I wanted to redo the complete experiement today but kilana was out of reach. I will retry locally and if it works I will publish that on 39a
should I use your script instead of clicking on the merge button? Setf
The merge button should work - but it sounds like there was some problem with your repository configuration. Simplest would be to make sure that all the relevant .mcz files are in your package- cache directory before you start the merge. Avi
I redid the experienc using another vm and it worked. Now I got an error when I pressed allNewer since I want all the new changes. fullTimeStamp not understood because
isLocalNewer ....self remoteDefinition fullTimeStamp
is nil
By the way what is source same but rev changed.....
What is rev? revision? What is a revision? ....uhhhhh
Stef
On 21 sept. 05, at 12:10, stéphane ducasse wrote:
Now I did the following:
Loaded morphic splitters added all the packages as required packages of Sqeuak and saved
then took back a new image, loaded MC274 and loaded the squeak.mcz file obtained previously and it froze squeak.
I'm redoing the experiment
Stef
On 21 sept. 05, at 08:20, Daniel Vainsencher wrote:
I had a bit of that when I was doing it, in my case because some packages were needed from source.sqf.org which was inaccessible at the time.
Discover what packages/versions it wants, and make sure you have repositories with those in the relevant repository group (this was the quick solution for me, switched to source2), or do like Avi says and get everything before hand.
Daniel
Avi Bryant wrote:
On Sep 20, 2005, at 10:53 PM, stéphane ducasse wrote:
Hi
yesterday I loaded the new version of MC and selected my old squeak. 2.mcz that was not loading previously and I did merge....
It works until I got an error like cannot find files....and I could not understand why. So I wanted to redo the complete experiement today but kilana was out of reach. I will retry locally and if it works I will publish that on 39a
should I use your script instead of clicking on the merge button? Setf
The merge button should work - but it sounds like there was some problem with your repository configuration. Simplest would be to make sure that all the relevant .mcz files are in your package-cache directory before you start the merge. Avi
Am 21.09.2005 um 12:43 schrieb stéphane ducasse:
I redid the experienc using another vm and it worked. Now I got an error when I pressed allNewer since I want all the new changes. fullTimeStamp not understood because
isLocalNewer ....self remoteDefinition fullTimeStamp
is nil
I guess nobody ever uses these newer/older buttons (timestamps are unreliable), so there might well be bugs in there. If you actually wanted to apply "keep" to all conflicts that's what the "rest remote" button does ... ("keep rest" would surely be a better wording).
By the way what is source same but rev changed.....
What is rev? revision? What is a revision? ....uhhhhh
It means the method has a different timestamp but the source code did not change. This happens, for example, when you insert a "self halt" while debugging and then delete the line instead of reverting to the previous version.
- Bert -
I guess nobody ever uses these newer/older buttons (timestamps are unreliable), so there might well be bugs in there. If you actually wanted to apply "keep" to all conflicts that's what the "rest remote" button does ... ("keep rest" would surely be a better wording).
But I have rest local and rest remote....So I guess that it should be rest remote?
By the way what is source same but rev changed.....
What is rev? revision? What is a revision? ....uhhhhh
It means the method has a different timestamp but the source code did not change. This happens, for example, when you insert a "self halt" while debugging and then delete the line instead of reverting to the previous version.
ok I see. mainly recompiling the method.
Still fighting with the morphic splitters ....working on code that is not yours and take ages to be proceed is a pain with tools with bad UI and the UI of the MC browser really sucks.
What is the difference between underline, bold, bold underlined? I'm restarting again the complete loading and importing to be sure.
Stef
Am 22.09.2005 um 12:44 schrieb stéphane ducasse:
Still fighting with the morphic splitters ....working on code that is not yours and take ages to be proceed is a pain with tools with bad UI and the UI of the MC browser really sucks.
What is the difference between underline, bold, bold underlined? I'm restarting again the complete loading and importing to be sure.
(from MCConflict>>summary)
self isResolved ifTrue: [self remoteChosen ifTrue: [#underlined] ifFalse: [#struckOut]] ifFalse: [#bold].
Bold means an unresolved conflict. You have to resolve all conflicts manually, choosing either the local ("reject") or remote ("keep") version.
If you choose to keep a change, it's underlined, otherwise, it's struck out.
- Bert -
On 22 sept. 05, at 13:32, Bert Freudenberg wrote:
Am 22.09.2005 um 12:44 schrieb stéphane ducasse:
Still fighting with the morphic splitters ....working on code that is not yours and take ages to be proceed is a pain with tools with bad UI and the UI of the MC browser really sucks.
What is the difference between underline, bold, bold underlined? I'm restarting again the complete loading and importing to be sure.
(from MCConflict>>summary)
self isResolved ifTrue: [self remoteChosen ifTrue: [#underlined]ifFalse: [#struckOut]] ifFalse: [#bold].
Bold means an unresolved conflict. You have to resolve all conflicts manually, choosing either the local ("reject") or remote ("keep") version.
If you choose to keep a change, it's underlined, otherwise, it's struck out.
thanks bert I was talking about the package itself. I was confused because after the save of the morphic splitter and reload in a fresh image and resolution of the merge, I could not really understand why some packages were underlined and other struck. Since this is not my code and it takes so long, it is difficult to learn from the actions.
Now I was restarting to redo everything sqeuak package with required packages saving loading morphic splitters from cs adding new packages as required saving squeak and I got yet another strange bug....
So I have to go back home and I hope to be able to find time to redo again the experiment I want to make sure of the process before pushing that in 3.9a
Stef
On 18 sept. 05, at 21:39, Andreas Raab wrote:
Hi -
The implications with this setup is that we have to produce often an image (validate a configuration of changes working together) and
this can be quite painful. At least they are a lot of try and error
when there are a lot of dependencies between packages.
In my experience that's a sign that a) you haven't found a good package structure and/or b) you don't test early enough. In the first case the effect is that higher-level changes get intermangled with lower-level changes (that do in fact require careful folding into a running system).
Not sure we got a problem with deltas since we got a change that removed a classVar on compiled method and the diff was not in sync.
An indication towards that point is the effect that the Kernel package has (I think) the most revisions, where clearly it is one of the most stable packages (if it isn't we're in big trouble) and should have the least number of modifications. The same is probably true for Morphic at this point - you *need* a better package structure for Morphic which makes clear when you touch the critical parts (requiring very careful testing) and when the non-critical ones.
Yes but this is exactly what we were doing with the introduction of morphic splitters. Now I simply cannot load the morphic splitters changes since they move classes around and introduced packages. I think that the script idea is not good for package restructuring. Because the morphic splitters is not complex but still finding the right order is terrible. Squeak died on me when I tried to load Morphic-aftermorphicsplitters. And I could NOT try sooner since we got a large set of changes from the morphicsplitter team.
In comparison, in Tweak I can typically tell by just looking at the package name whether this will require any effort to make sure it loads correctly or not.
I imagine. Once your system is well factored and if you do not change the structure it should work.
It is also really important to test "early enough" whether you can load a package or not after you've done a modification. Unfortunately, you guys are somewhat handicapped since you don't have an automatic default update mechanism like we're using in Tweak. When I post a new version of a package that has even the slightest chance to affect anything I immediately get a fresh image, hit update and test it.
this was what I did with Envy... Joseph always told me that you have only finished when you can load from a fresh image. I do the same with store. Now previously this was easier since this is my code :) I know it and this is easier and smaller.
Cost me perhaps twenty seconds but if it doesn't work I can easily either post an appropriate configuration or make a change to the package to make it load and post a new version of the package. I really couldn't imagine to work like you do,
me too so this is why it should change or 3.9 is stuck. Nobody will want to do that.
e.g., make a whole set of new package versions, try to load and figure out where and why it dies.
But how can we do something else?
What I'm trying now is to use MC fully. So I did a squeak package added all the packages as dependent saved. loaded all the morphic splitters updated the squeak required packages saved and now I'm trying to load from the latest image to see if this is working this is just plain slow and I'm waiting to see the result.
I hope that this is good. Because I think that this is the right way to use MC.
Bert I have the impression that using package dependencies with a
root package (ie havgin Squeak and all the image package as required) would help to minimize inter package dependencies, now why do you use your script (the explicit list of packages)
instead of required package. Is it because you can push that in the
update stream?
Early on we looked at using dependencies but they didn't work very well for what we wanted to do. Most importantly, if you have a newer version of a package it will load the older version instead (VERY bad).
I think that this is disconnected. The process should be - load the latest build - check most recent package (or the build should always be the most recent ie. we do not publish individual packages)
Avi? What is your point of view on that.?
There were also some serious issues with nested dependencies (Bert and I had a lot of fun figuring that out).
I think that we should not use nested dependencies.
And keep in mind that dependencies are not as good as a load-order since they typically only specify a partial relation (A dependsOn: X and B dependsOn: X does not specify whether A is loaded before B or vice versa) whereas we wanted to make sure that the updating process uses a single well-defined load order for everyone.
I see. But right now there are situation where I simply cannot load at all.
stéphane ducasse wrote:
Yes but this is exactly what we were doing with the introduction of morphic splitters. Now I simply cannot load the morphic splitters changes since they move classes around and introduced packages. I think that the script idea is not good for package restructuring. Because the morphic splitters is not complex but still finding the right order is terrible. Squeak died on me when I tried to load Morphic-aftermorphicsplitters. And I could NOT try sooner since we got a large set of changes from the morphicsplitter team.
A good question is whether Daniel's proposed mechanism would be able to deal with this problem.
e.g., make a whole set of new package versions, try to load and figure out where and why it dies.
But how can we do something else?
Well, generally speaking, having the ability to do a quick check if some changes will load okay is certainly critical even if one insists on posting configurations explicitly (for which I don't see much of a reason).
For something like splitters change set it's probably worthwhile to treat it as a goal and have someone work on figuring out a load-order before trying to fold it in (unless Daniel's simu-load is capable to deal with it).
Early on we looked at using dependencies but they didn't work very well for what we wanted to do. Most importantly, if you have a newer version of a package it will load the older version instead (VERY bad).
I think that this is disconnected. The process should be
- load the latest build
- check most recent package (or the build should always be the most
recent ie. we do not publish individual packages)
Avi? What is your point of view on that.?
Actually, there is something to be said for having a tool to deal with updates. What it might do is list all the (dependent) packages that are associated with a (master) package and show you which packages are older / up to date / newer than in the configuration. The action to take (upgrade, downgrade, leave alone) could be covered by preferences (e.g., "never downgrade", "always load exact version"...).
Add to this a way to "update all dependent packages" (which means go out and see if there are any newer dependent packages for a master package) and I think we've got all the bases covered. Explicit updates for cautious people, implicit updates for the adventurous folks.
Cheers, - Andreas
packages@lists.squeakfoundation.org
-
Andreas Raab -
Avi Bryant -
Bert Freudenberg -
Colin Putney -
Daniel Vainsencher -
stéphane ducasse