Hi Chris, there´s no problem at all with adding the package code anywhere you want. Feel free to contact me for any suggestion.
From what I´ve seen from the sample code I sent to you, although the image
size grows up, when I closed the repository and evaluated MagmaSession cleanUp, the image size goes back to the state before the evaluation. In the "real migration" what I told in the first mail still the size is growing at very high rates, we cannot detect yet what objects aren´t collected by the GC.
Regards, Juan Matías.
On 8/10/07, Chris Muller ma.chris.m@gmail.com wrote:
Hi Juan, THANKS for the best test case code anyone has ever sent me for asking a question about Magma..!
I have been too busy to get to it but am finally spending some time with this tonight.
Do you mind if I harvest this for the Magma tester package? It might be a really great addition for benchmarking / profiling memory as well overall performance.
Let me explore this; I'll write again in the next few days..
Regards, Chris
On 8/7/07, Burella Juan M. juan.burella@gmail.com wrote:
Hi Chris, Thanks for your answer. I've attached a monticello package with the tests, open a
Transcript
window and see class comments for instructions of how to run them.
This is the Transcript output for 50 "trees" with 5 of depth:
Creating magma repository Opening magma session Image size before creating magma collection: 69851252 Creating magma collection at MagmaMemoryTest --Iteration: 1..5: Image size BEFORE creating test object: 69851252 Creating test objects... Image size AFTER creating test object: 69851252 Saving 5 next objects from collection to magma collection... Image size BEFORE Magma cleaning: 85098444 Image size AFTER Magma cleaning: 82623944 --Iteration: 6..10: Image size BEFORE creating test object: 82623944 Creating test objects... Image size AFTER creating test object: 82623944 Saving 5 next objects from collection to magma collection... Image size BEFORE Magma cleaning: 106407888 Image size AFTER Magma cleaning: 83066628 --Iteration: 11..15: Image size BEFORE creating test object: 83066628 Creating test objects... Image size AFTER creating test object: 83066628 Saving 5 next objects from collection to magma collection... Image size BEFORE Magma cleaning: 108570520 Image size AFTER Magma cleaning: 85184212 --Iteration: 16..20: Image size BEFORE creating test object: 85184212 Creating test objects... Image size AFTER creating test object: 85184212 Saving 5 next objects from collection to magma collection... Image size BEFORE Magma cleaning: 108627824 Image size AFTER Magma cleaning: 85212848 --Iteration: 21..25: Image size BEFORE creating test object: 85212848 Creating test objects... Image size AFTER creating test object: 85212848 Saving 5 next objects from collection to magma collection... Image size BEFORE Magma cleaning: 108615528 Image size AFTER Magma cleaning: 85216916 --Iteration: 26..30: Image size BEFORE creating test object: 85216916 Creating test objects... Image size AFTER creating test object: 85216916 Saving 5 next objects from collection to magma collection... Image size BEFORE Magma cleaning: 108787536 Image size AFTER Magma cleaning: 85257868 --Iteration: 31..35: Image size BEFORE creating test object: 85257868 Creating test objects... Image size AFTER creating test object: 85257868 Saving 5 next objects from collection to magma collection... Image size BEFORE Magma cleaning: 108857164 Image size AFTER Magma cleaning: 85307012 --Iteration: 36..40: Image size BEFORE creating test object: 85307012 Creating test objects... Image size AFTER creating test object: 85307012 Saving 5 next objects from collection to magma collection... Image size BEFORE Magma cleaning: 108623676 Image size AFTER Magma cleaning: 85257812 --Iteration: 41..45: Image size BEFORE creating test object: 85257812 Creating test objects... Image size AFTER creating test object: 85257812 Saving 5 next objects from collection to magma collection... Image size BEFORE Magma cleaning: 108390116 Image size AFTER Magma cleaning: 85159312 --Iteration: 46..50: Image size BEFORE creating test object: 85159312 Creating test objects... Image size AFTER creating test object: 85159312 Saving 5 next objects from collection to magma collection... Image size BEFORE Magma cleaning: 108594588 Image size AFTER Magma cleaning: 85183668
On 8/7/07, Chris Muller asqueaker@gmail.com wrote:
HI Juan,
I really need to add a "How to do a Bulk Load" page to the
documentation,
sorry.
In the meantime, you weren't clear what you were adding the 19K trees of objects *TO*. My guess is you are adding to a Smalltlak collection; a Dictionary or OrderedCollection perhaps? It's good you are doing the stubbing and finalizieOids, but they won't help much if the root collection is a Smalltalk collection because the entire persistent graph is in memory then (except for the stubbed, maybe).
You need to add them to a MagmaCollection instead, so that transparent paging of objects in and out of memory can occur.
To verify what kinds of persistents you have in memory, check out your #cachedObjectCountByClass. You should have no more than one 19K tree worth of objects given that you are doing a garbageCollect after each one.
If you would share the skeleton of your load script it also might shed some light.
Thanks, Chris
On 8/6/07, Burella Juan M. juan.burella@gmail.com wrote:
I need to do a massive migration to Magma. My approach is currently
like
this: I'm loading the data from external files, then build a complex
object
and commit it to a magma repository, and over again until the file
ends.
While doing this, my image was growing at very high rates, approx.
200Mb
and
taking several days, between 5 or 6. So, to see what's happening,
I've
done
some tests with toy objects, each one of them has 5 iVars. and each
one
has
another 5 iVars. This generates a tree of 19.556 objects (5^0+5^1+5^2+5^3+5^4+5^5). Then I started to adding 50 of these
"tree"
(or
smock?) objects, by groups of 5. I tried to maintain the complexity
of
these
objects by adding "random" variable sized strings to the leafs and
other
miscellaneous objets. When this 50 are finalized, I've done:
self magmaSession stubOut: aCollection. "aCollection has 50 smock
objects"
aCollection := nil. Smalltalk garbageCollect. self magmaSession finalizeOids.
But still the image is increasing its size. At beggining of the test
the
image size (as reported from #vmParameterAt: 3) was approx. 60Mb and
at
the
end was approx. 89MB.
Any suggestions for reducing the growing of the image size?
Thanks in advance. Juan Matias.
Magma mailing list Magma@lists.squeakfoundation.org
http://lists.squeakfoundation.org/mailman/listinfo/magma
Juan, I spent some time running your neat test case but it didn't show me any unreasonable growth. So I then went back to the output you posted and looked at it in detail.
Regarding the output you pasted in your previous message:
======================================= Creating magma repository Opening magma session Image size before creating magma collection: 69851252 Creating magma collection at MagmaMemoryTest --Iteration: 1..5: Image size BEFORE creating test object: 69851252 Creating test objects... Image size AFTER creating test object: 69851252 Saving 5 next objects from collection to magma collection... Image size BEFORE Magma cleaning: 85098444 Image size AFTER Magma cleaning: 82623944 --Iteration: 6..10: Image size BEFORE creating test object: 82623944 Creating test objects... Image size AFTER creating test object: 82623944 Saving 5 next objects from collection to magma collection... Image size BEFORE Magma cleaning: 106407888 Image size AFTER Magma cleaning: 83066628 ... --Iteration: 46..50: Image size BEFORE creating test object: 85159312 Creating test objects... Image size AFTER creating test object: 85159312 Saving 5 next objects from collection to magma collection... Image size BEFORE Magma cleaning: 108594588 Image size AFTER Magma cleaning: 85183668 ==================================
End of memory at the beginning shows:
--Iteration: 6..10: Image size BEFORE creating test object: 82623944
at the end:
Image size AFTER Magma cleaning: 85183668
85183668 -82623944 = 2559724
So, end-of-memory, through all of the bulk-load operations, needed to advance only 2.5MB through dynamic allocation, etc. Is this unreasonable?
Further, your output indicates there was only initial mild growth from the first three iterations. Then, beginning with --Iteration: 16..20:, it stayed relatively static at roughly 85MB.
Am I missing something?
Regards, Chris
On 8/10/07, Burella Juan M. juan.burella@gmail.com wrote:
Hi Chris, there´s no problem at all with adding the package code anywhere you want. Feel free to contact me for any suggestion. From what I´ve seen from the sample code I sent to you, although the image size grows up, when I closed the repository and evaluated MagmaSession cleanUp, the image size goes back to the state before the evaluation. In the "real migration" what I told in the first mail still the size is growing at very high rates, we cannot detect yet what objects aren´t collected by the GC.
Regards, Juan Matías.
On 8/10/07, Chris Muller ma.chris.m@gmail.com wrote:
Hi Juan, THANKS for the best test case code anyone has ever sent me for asking a question about Magma..!
I have been too busy to get to it but am finally spending some time with this tonight.
Do you mind if I harvest this for the Magma tester package? It might be a really great addition for benchmarking / profiling memory as well overall performance.
Let me explore this; I'll write again in the next few days..
Regards, Chris
On 8/7/07, Burella Juan M. < juan.burella@gmail.com> wrote:
Hi Chris, Thanks for your answer. I've attached a monticello package with the tests, open a
Transcript
window and see class comments for instructions of how to run them.
This is the Transcript output for 50 "trees" with 5 of depth:
Creating magma repository Opening magma session Image size before creating magma collection: 69851252 Creating magma collection at MagmaMemoryTest --Iteration: 1..5: Image size BEFORE creating test object: 69851252 Creating test objects... Image size AFTER creating test object: 69851252 Saving 5 next objects from collection to magma collection... Image size BEFORE Magma cleaning: 85098444 Image size AFTER Magma cleaning: 82623944 --Iteration: 6..10: Image size BEFORE creating test object: 82623944 Creating test objects... Image size AFTER creating test object: 82623944 Saving 5 next objects from collection to magma collection... Image size BEFORE Magma cleaning: 106407888 Image size AFTER Magma cleaning: 83066628 --Iteration: 11..15: Image size BEFORE creating test object: 83066628 Creating test objects... Image size AFTER creating test object: 83066628 Saving 5 next objects from collection to magma collection... Image size BEFORE Magma cleaning: 108570520 Image size AFTER Magma cleaning: 85184212 --Iteration: 16..20: Image size BEFORE creating test object: 85184212 Creating test objects... Image size AFTER creating test object: 85184212 Saving 5 next objects from collection to magma collection... Image size BEFORE Magma cleaning: 108627824 Image size AFTER Magma cleaning: 85212848 --Iteration: 21..25: Image size BEFORE creating test object: 85212848 Creating test objects... Image size AFTER creating test object: 85212848 Saving 5 next objects from collection to magma collection... Image size BEFORE Magma cleaning: 108615528 Image size AFTER Magma cleaning: 85216916 --Iteration: 26..30: Image size BEFORE creating test object: 85216916 Creating test objects... Image size AFTER creating test object: 85216916 Saving 5 next objects from collection to magma collection... Image size BEFORE Magma cleaning: 108787536 Image size AFTER Magma cleaning: 85257868 --Iteration: 31..35: Image size BEFORE creating test object: 85257868 Creating test objects... Image size AFTER creating test object: 85257868 Saving 5 next objects from collection to magma collection... Image size BEFORE Magma cleaning: 108857164 Image size AFTER Magma cleaning: 85307012 --Iteration: 36..40: Image size BEFORE creating test object: 85307012 Creating test objects... Image size AFTER creating test object: 85307012 Saving 5 next objects from collection to magma collection... Image size BEFORE Magma cleaning: 108623676 Image size AFTER Magma cleaning: 85257812 --Iteration: 41..45: Image size BEFORE creating test object: 85257812 Creating test objects... Image size AFTER creating test object: 85257812 Saving 5 next objects from collection to magma collection... Image size BEFORE Magma cleaning: 108390116 Image size AFTER Magma cleaning: 85159312 --Iteration: 46..50: Image size BEFORE creating test object: 85159312 Creating test objects... Image size AFTER creating test object: 85159312 Saving 5 next objects from collection to magma collection... Image size BEFORE Magma cleaning: 108594588 Image size AFTER Magma cleaning: 85183668
On 8/7/07, Chris Muller asqueaker@gmail.com wrote:
HI Juan,
I really need to add a "How to do a Bulk Load" page to the
documentation,
sorry.
In the meantime, you weren't clear what you were adding the 19K trees of objects *TO*. My guess is you are adding to a Smalltlak collection; a Dictionary or OrderedCollection perhaps? It's good you are doing the stubbing and finalizieOids, but they won't help much if the root collection is a Smalltalk collection because the entire persistent graph is in memory then (except for the stubbed, maybe).
You need to add them to a MagmaCollection instead, so that transparent paging of objects in and out of memory can occur.
To verify what kinds of persistents you have in memory, check out your #cachedObjectCountByClass. You should have no more than one 19K tree worth of objects given that you are doing a garbageCollect after each one.
If you would share the skeleton of your load script it also might shed some light.
Thanks, Chris
On 8/6/07, Burella Juan M. juan.burella@gmail.com wrote:
I need to do a massive migration to Magma. My approach is currently
like
this: I'm loading the data from external files, then build a complex
object
and commit it to a magma repository, and over again until the file
ends.
While doing this, my image was growing at very high rates, approx.
200Mb
and
taking several days, between 5 or 6. So, to see what's happening,
I've
done
some tests with toy objects, each one of them has 5 iVars. and each
one
has
another 5 iVars. This generates a tree of 19.556 objects (5^0+5^1+5^2+5^3+5^4+5^5). Then I started to adding 50 of these
"tree"
(or
smock?) objects, by groups of 5. I tried to maintain the complexity
of
these
objects by adding "random" variable sized strings to the leafs and
other
miscellaneous objets. When this 50 are finalized, I've done:
self magmaSession stubOut: aCollection. "aCollection has 50 smock
objects"
aCollection := nil. Smalltalk garbageCollect. self magmaSession finalizeOids.
But still the image is increasing its size. At beggining of the test
the
image size (as reported from #vmParameterAt: 3) was approx. 60Mb and
at
the
end was approx. 89MB.
Any suggestions for reducing the growing of the image size?
Thanks in advance. Juan Matias.
Magma mailing list Magma@lists.squeakfoundation.org
http://lists.squeakfoundation.org/mailman/listinfo/magma
Magma mailing list Magma@lists.squeakfoundation.org http://lists.squeakfoundation.org/mailman/listinfo/magma
Hi Chris, sorry for the delayed reply; I'm going to test the sample trying to aproximate the depth of my real objects, which could be of 20 levels per object, having 12,000 of them (if you remember from the first mail I´m building these objects one by one collecting the data from a file). Under this scenario, the image growth reach 200Mbytes and it became unstable, so I have to shutdown the image and restart again all the process. I cannot reproduce this scenario with the samples I´ve sent to you, and I think that´s the reason why you haven´t seen the problem we're experimenting here. Now we´re going to test the sample with the numbers above, and if I can reproduce the behavoir I´ll send to you the results.
Thanks Juan Matías.
On 8/10/07, Chris Muller asqueaker@gmail.com wrote:
Juan, I spent some time running your neat test case but it didn't show me any unreasonable growth. So I then went back to the output you posted and looked at it in detail.
Regarding the output you pasted in your previous message:
======================================= Creating magma repository Opening magma session Image size before creating magma collection: 69851252 Creating magma collection at MagmaMemoryTest --Iteration: 1..5: Image size BEFORE creating test object: 69851252 Creating test objects... Image size AFTER creating test object: 69851252 Saving 5 next objects from collection to magma collection... Image size BEFORE Magma cleaning: 85098444 Image size AFTER Magma cleaning: 82623944 --Iteration: 6..10: Image size BEFORE creating test object: 82623944 Creating test objects... Image size AFTER creating test object: 82623944 Saving 5 next objects from collection to magma collection... Image size BEFORE Magma cleaning: 106407888 Image size AFTER Magma cleaning: 83066628 ... --Iteration: 46..50: Image size BEFORE creating test object: 85159312 Creating test objects... Image size AFTER creating test object: 85159312 Saving 5 next objects from collection to magma collection... Image size BEFORE Magma cleaning: 108594588 Image size AFTER Magma cleaning: 85183668 ==================================
End of memory at the beginning shows:
--Iteration: 6..10: Image size BEFORE creating test object: 82623944
at the end:
Image size AFTER Magma cleaning: 85183668
85183668 -82623944 = 2559724
So, end-of-memory, through all of the bulk-load operations, needed to advance only 2.5MB through dynamic allocation, etc. Is this unreasonable?
Further, your output indicates there was only initial mild growth from the first three iterations. Then, beginning with --Iteration: 16..20:, it stayed relatively static at roughly 85MB.
Am I missing something?
Regards, Chris
magma@lists.squeakfoundation.org