Hi!
Ok, so Gjallar is improving day-by-day and the current Gjallar is built on Squeak 3.10.2-7179 and MC packages "r42Alpha4" + "Magma sunit" from MagmaTester on SS. I have been working on this latest incarnation of Magma the latest few days and it seems to be working fine (although marked as alpha).
I now need to migrate an existing Gjallar customer over to the above new system/version.
The older Gjallar runs MagmaServerLoader-kph.35 in a 3.8.1-6747 image. This is of course a rather old Magma but it has been running for over a year, continuous use (although on the lighter side) with no serious problems at all. Good work Chris! :)
Questions:
1. Would I be better off to use r41.2 than r42Alpha? Presuming that is the latest official release, which I think it is?
2. Are there migration instructions?
I did the simplest thing first, just opened up the old db using the new image. Of course it says that hey, this repo is version 9 so I need to migrate it. Cool, go ahead. :)
Then after that phase I stumbled on 2 issues I "solved":
- Missing ivar in MaRootAnchor. Fine, seems to be "stats" so I just told Magma to go forward with that.
- Then Magma blew up because it was sending "node" in #sendToWarmBackups: and getting nil. After a bit of investigation I realized this is related to the latest HA-code so this little trick seemed to repair that:
MagmaRepositoryController allInstances anyOne session commit: [node := MagmaNode new]
Then it got harder. :) It looks very promising at this stage, most of Gjallar seems to work fine. I can explore the db and it all looks fine except for one important thing: SortedCollections with stored sortBlocks
In the old image it all works fine, in the new image I end up with a garbled SortedCollection with these ivars: array: 4081353 firstIndex: 281473902968834 lastIndex: 281473902968835 sortBlock: 4081234
...so obviously there is a problem here with SortedCollection instances *that have sortBlocks* - because in other places I have SortedCollections with a nil sortBlock, and those work fine.
Hmmm, I will try to work around this but of course it might be interesting for you to be aware of the issue.
regards, Göran
Hi again!
Status report:
I managed to brute force myself through the problem with SortedCollection by converting them to OCs in the old db/image, then opening it up in the new image and converting back. So that is fixed.
Now I am facing a different issue - I have instances of Q2User (the user object in Gjallar) that now needs to "become" instance of Q2PersonalUser - a subclass of Q2User. Same ivars, just a refactoring of the hierarchy.
...ehm... do I need to use MagmaFileTraverser or does Magma have some nice hooks for picking a different class?
regards, Göran
Hi Göran, I recommend using the latest r42 alpha code as you have.
I'm not sure what happened with the SortedCollections, but I'm glad you got it fixed.
Unfortunately, Magma does NOT support become through the session, because that would require knowing all references to an object in the huge repository. But yes, the MagmaFileTraverser could be useful for performing such a migration..
- Chris
On Thu, Feb 26, 2009 at 9:48 AM, Göran Krampe goran@krampe.se wrote:
Hi again!
Status report:
I managed to brute force myself through the problem with SortedCollection by converting them to OCs in the old db/image, then opening it up in the new image and converting back. So that is fixed.
Now I am facing a different issue - I have instances of Q2User (the user object in Gjallar) that now needs to "become" instance of Q2PersonalUser - a subclass of Q2User. Same ivars, just a refactoring of the hierarchy.
...ehm... do I need to use MagmaFileTraverser or does Magma have some nice hooks for picking a different class?
regards, Göran
Hi!
Chris Muller wrote:
Hi Göran, I recommend using the latest r42 alpha code as you have.
Good! :)
I'm not sure what happened with the SortedCollections, but I'm glad you got it fixed.
Yeah, got past that.
Unfortunately, Magma does NOT support become through the session, because that would require knowing all references to an object in the huge repository. But yes, the MagmaFileTraverser could be useful for performing such a migration..
It turned out primitiveChangeClassTo: worked fine! I will blog about the details. It would also be nice to have more info on available hooks for doing migration, or add such hooks. :) And oh, I get this truncation warning - I understand it, but how do I make them disappear? Or in other words - how do I get Magma to grok the new class shape? :)
regards, Göran
It turned out primitiveChangeClassTo: worked fine! I will blog about the details. It would also be nice to have more info on available hooks for doing migration, or add such hooks. :)
Hmmm.. I'm not sure how Magma will respond to that. It *seems* like it should be fine because it should pick up the new classId and commit. But please do proceed with caution going forward and make sure all is well before deploying this model! I'm not sure I have enough information give exact advice but here is how Magma works so you may be able to determine the proper course.
- As part of the repository Definition, there is a Dictionary of "classDefinitionsById", keyed by the classId, valued by an OrderedCollection of MaClassDefinitions. - Each MaClassDefinition is for a particular shape of that class name, shape defined by its "allInstVarNames". When some session commits an instance of that class with different set of instVarNames, an additional MaClassDefinition is added to the OrderedCollection at that classId. - Every object buffer knows its #classId and #classVersion, identifying the exact shape of the object.
What you want to do is change each Q2User object-buffer to point to the proper version (probably 1) of the new classId for Q2PersonalUser. The best way is to update your references to Q2User to new instances of Q2PersonalUser, but I understand this may not be practical.
And oh, I get this truncation warning
- I understand it, but how do I make them disappear? Or in other words - how
do I get Magma to grok the new class shape? :)
The truncation warning occurs when the buffer for an object downloaded into an image from the server (i.e., via a proxy-mat read request) has instVars names that are not present for that class in that image. IOW, the *buffer* has a link to an object that the client image has no slot for. This means a reduced MaClassDefinition for that "shape" of object will also be created if not already present if change to that object is committed. Once the object is committed, the ObjectBuffer will refer to the new classVersion and the conditions that generate the warning will no longer be present the next time.
Magma handles class-shape changes and renames "out of the box" but more complex migrations such as hierarchy refactorings must be addressed on a case-by-case basis, like what you have here.
- Chris
On Tue, Mar 3, 2009 at 3:38 PM, Göran Krampe goran@krampe.se wrote:
Hi!
Chris Muller wrote:
Hi Göran, I recommend using the latest r42 alpha code as you have.
Good! :)
I'm not sure what happened with the SortedCollections, but I'm glad you got it fixed.
Yeah, got past that.
Unfortunately, Magma does NOT support become through the session, because that would require knowing all references to an object in the huge repository. But yes, the MagmaFileTraverser could be useful for performing such a migration..
It turned out primitiveChangeClassTo: worked fine! I will blog about the details. It would also be nice to have more info on available hooks for doing migration, or add such hooks. :) And oh, I get this truncation warning
- I understand it, but how do I make them disappear? Or in other words - how
do I get Magma to grok the new class shape? :)
regards, Göran
Magma mailing list Magma@lists.squeakfoundation.org http://lists.squeakfoundation.org/mailman/listinfo/magma
Hi!
Chris Muller wrote:
It turned out primitiveChangeClassTo: worked fine! I will blog about the details. It would also be nice to have more info on available hooks for doing migration, or add such hooks. :)
Hmmm.. I'm not sure how Magma will respond to that. It *seems* like it should be fine because it should pick up the new classId and commit. But please do proceed with caution going forward and make sure all is well before deploying this model! I'm not sure I have enough information give exact advice but here is how Magma works so you may be able to determine the proper course.
- As part of the repository Definition, there is a Dictionary of
"classDefinitionsById", keyed by the classId, valued by an OrderedCollection of MaClassDefinitions.
- Each MaClassDefinition is for a particular shape of that class name,
shape defined by its "allInstVarNames". When some session commits an instance of that class with different set of instVarNames, an additional MaClassDefinition is added to the OrderedCollection at that classId.
- Every object buffer knows its #classId and #classVersion,
identifying the exact shape of the object.
I will take a look at the repo definition, how can I "inspect" the object buffers in an easy way? It really would be cool to have some kind of low level explorer that does NOT deserialize/serialize these guys.
What you want to do is change each Q2User object-buffer to point to the proper version (probably 1) of the new classId for Q2PersonalUser.
Right.
The best way is to update your references to Q2User to new instances of Q2PersonalUser, but I understand this may not be practical.
Right. :)
And oh, I get this truncation warning
- I understand it, but how do I make them disappear? Or in other words - how
do I get Magma to grok the new class shape? :)
The truncation warning occurs when the buffer for an object downloaded into an image from the server (i.e., via a proxy-mat read request) has instVars names that are not present for that class in that image. IOW, the *buffer* has a link to an object that the client image has no slot for. This means a reduced MaClassDefinition for that "shape" of object will also be created if not already present if change to that object is committed. Once the object is committed, the ObjectBuffer will refer to the new classVersion and the conditions that generate the warning will no longer be present the next time.
Yes, but... let's say I have tons of objects in the db with this superfluous ivar in them. Every time I pull one of those guys into the image and changes it and commits it back - it will warn me. Fine. But how do I fix it "bulk wise"? I could iterate over them, but how do I tell Magma that:
"Yes, consider it modified, although it is not, but I want to commit and truncate all these guys now in order to not get warnings for the rest of my life..." :)
regards, Göran
Hi,
I will take a look at the repo definition, how can I "inspect" the object buffers in an easy way? It really would be cool to have some kind of low level explorer that does NOT deserialize/serialize these guys.
Ah, MagmaBufferController has been around since 2005 for this exact purpose! :-) It's in the "Magma Tools" package in the MagmaTester project of squeaksource. You may wish to make a simple UI to render the buffers, I made one in Maui in about 37 seconds.. :)
Yes, but... let's say I have tons of objects in the db with this superfluous ivar in them. Every time I pull one of those guys into the image and changes it and commits it back - it will warn me. Fine. But how do I fix it "bulk wise"? I could iterate over them, but how do I tell Magma that:
"Yes, consider it modified, although it is not, but I want to commit and truncate all these guys now in order to not get warnings for the rest of my life..." :)
Heh, ok, there are two separate issues here, the warning and the truncation. Yes, I had to look up whether a truncation is considered a "change" and it appears it is not. But this is actually a good thing; or at least the "more conservative" thing.
It sounds like you only care about truncation to the extent it rids the warning. Can't you just handle and resume that where you perform your commit? Perhaps you have commits all over the code? Hmm.. If so, I suppose you could extend MagmaTruncationWarning with:
defaultAction self resume
or some such..?
Hi!
Chris Muller wrote:
Hi,
I will take a look at the repo definition, how can I "inspect" the object buffers in an easy way? It really would be cool to have some kind of low level explorer that does NOT deserialize/serialize these guys.
Ah, MagmaBufferController has been around since 2005 for this exact purpose! :-) It's in the "Magma Tools" package in the MagmaTester project of squeaksource. You may wish to make a simple UI to render the buffers, I made one in Maui in about 37 seconds.. :)
Ah, nice. Will take a look.
Yes, but... let's say I have tons of objects in the db with this superfluous ivar in them. Every time I pull one of those guys into the image and changes it and commits it back - it will warn me. Fine. But how do I fix it "bulk wise"? I could iterate over them, but how do I tell Magma that:
"Yes, consider it modified, although it is not, but I want to commit and truncate all these guys now in order to not get warnings for the rest of my life..." :)
Heh, ok, there are two separate issues here, the warning and the truncation. Yes, I had to look up whether a truncation is considered a "change" and it appears it is not. But this is actually a good thing; or at least the "more conservative" thing.
Yes.
It sounds like you only care about truncation to the extent it rids the warning. Can't you just handle and resume that where you perform your commit? Perhaps you have commits all over the code? Hmm.. If so, I suppose you could extend MagmaTruncationWarning with:
defaultAction self resume
or some such..?
Well, I ended up doing just that - and no, the commits in Gjallar are not spread out. BUT... this means I ignore all truncation warnings from now on! Not exactly what I wanted to do... :)
regards, Go"ran
Hey, what do you think about upgrading MagmaTruncationWarning with an instVar, 'truncatedObjects' or something of your preference.
Then just update #warnOfTruncationsIn: to populate it. Then Gjaller can interrogate the objects and decide whether it wants to pass or resume the warning.
You are a developer in the MagmaTester project on SqueakSource. Let me know if you do it, I'll merge it in.
Regards, Chris
On Wed, Mar 4, 2009 at 1:10 AM, Göran Krampe goran@krampe.se wrote:
Hi!
Chris Muller wrote:
Hi,
I will take a look at the repo definition, how can I "inspect" the object buffers in an easy way? It really would be cool to have some kind of low level explorer that does NOT deserialize/serialize these guys.
Ah, MagmaBufferController has been around since 2005 for this exact purpose! :-) It's in the "Magma Tools" package in the MagmaTester project of squeaksource. You may wish to make a simple UI to render the buffers, I made one in Maui in about 37 seconds.. :)
Ah, nice. Will take a look.
Yes, but... let's say I have tons of objects in the db with this superfluous ivar in them. Every time I pull one of those guys into the image and changes it and commits it back - it will warn me. Fine. But how do I fix it "bulk wise"? I could iterate over them, but how do I tell Magma that:
"Yes, consider it modified, although it is not, but I want to commit and truncate all these guys now in order to not get warnings for the rest of my life..." :)
Heh, ok, there are two separate issues here, the warning and the truncation. Yes, I had to look up whether a truncation is considered a "change" and it appears it is not. But this is actually a good thing; or at least the "more conservative" thing.
Yes.
It sounds like you only care about truncation to the extent it rids the warning. Can't you just handle and resume that where you perform your commit? Perhaps you have commits all over the code? Hmm.. If so, I suppose you could extend MagmaTruncationWarning with:
defaultAction self resume
or some such..?
Well, I ended up doing just that - and no, the commits in Gjallar are not spread out. BUT... this means I ignore all truncation warnings from now on! Not exactly what I wanted to do... :)
regards, Go"ran
Magma mailing list Magma@lists.squeakfoundation.org http://lists.squeakfoundation.org/mailman/listinfo/magma
magma@lists.squeakfoundation.org